How feedback resolution affects sub-micron precision motion systems

A finer encoder resolution allows a motion controller to distinguish smaller position increments, but it does not by itself establish where the tool, workpiece, or inspection sensor is within a fraction of a micrometre. In sub-micron precision motion systems, the useful benefit of finer feedback depends on noise, sensor accuracy, measurement location, mechanical stiffness, servo behavior, and temperature stability.

The practical question is not simply how many nanometres correspond to one encoder count. It is whether the complete feedback and mechanical system can detect a meaningful position error, correct it without introducing instability, and maintain the required position under the conditions in which the machine operates.

Resolution is not the same as positioning accuracy

Feedback resolution is the smallest position increment represented by the measurement system’s digital output. For an encoder with a stated output increment of 0.01 µm, one count corresponds nominally to 10 nm of measured displacement. That number describes the digital scale; it does not mean that every reported position is accurate to 10 nm.

Several related specifications describe different aspects of performance:

• Resolution: the nominal size of a reported increment.
• Accuracy: the closeness of a measured or achieved position to its reference value, under stated conditions.
• Repeatability: the consistency with which a position can be measured or reached over repeated attempts, under specified conditions.
• Noise: variation in the reported signal that does not represent the desired position change.
• Stability: the extent to which the measured or controlled position remains unchanged over a defined period.

A scale can report 10 nm increments while its output fluctuates across several counts at rest. It can also have interpolation error or scale calibration error that is much larger than one count. Conversely, a system may return consistently to a position while retaining a systematic offset from the true position. Repeatability and accuracy therefore need separate evidence.

Specifications also need a reference. Encoder accuracy describes the sensor under its specified conditions; stage positioning accuracy describes a broader assembly; and process accuracy concerns the point where the operation takes place. These values should not be exchanged without checking what each measurement includes.

What a controller can do with finer feedback

A position servo compares a commanded position with measured position and uses the resulting error to influence actuator output. If the feedback increment is coarse relative to the required correction, small position changes can remain invisible until the measured value crosses a count boundary. Quantization can limit how smoothly the controller detects slow motion or settles near a target.

Finer resolution reduces that quantization interval. It can make small deviations observable and support smoother control, provided those deviations are distinguishable from measurement noise. In a system with 0.1 µm feedback increments, a nominal 0.05 µm movement is smaller than one increment; its representation depends on the initial position relative to the count boundaries. Reducing the increment to 0.01 µm provides more digital detail, but does not establish that the reported change is physically accurate or repeatable.

At standstill, excessive noise can cause the reported position to alternate between counts. A controller that responds too aggressively may command unnecessary actuator corrections, producing position dither, audible activity, or heat. Filtering can reduce the noise seen by the servo, but it also introduces delay or reduces the bandwidth over which position changes are passed to the controller. The appropriate balance depends on the mechanics and the required settling behavior.

Finer feedback can also expose mechanical resonances that were less visible with coarser measurements. Increasing servo gain merely because the encoder reports smaller increments is not a reliable route to better performance: a compliant stage, flexible coupling, or lightly damped structure can become unstable before the controller makes useful corrections at that scale.

Nominal counts versus usable information

Encoder outputs are often derived by interpolating a periodic sensor signal into smaller subdivisions. A higher interpolation factor produces a smaller nominal count size, but the underlying signal quality still matters. Electrical noise, signal amplitude or offset variation, contamination, and interpolation nonlinearity can limit the accuracy of the subdivisions.

For a meaningful assessment, examine the position signal at rest as well as during motion. A stationary-axis noise trace should identify the measurement duration, sampling conditions, filtering, and whether values are expressed as peak-to-peak, root-mean-square, or another statistic. A peak-to-peak figure from a short observation cannot be compared directly with an RMS value from a different test.

Velocity estimation adds another constraint. When velocity is calculated from changes in position, finer quantization can improve the representation of low-speed motion, but differentiation also makes high-frequency position noise more visible. Controller sampling rate, encoder update rate, communication latency, and filtering therefore influence the benefit of a finer output increment.

Interface compatibility matters as much as the encoder’s headline specification. The controller must accept the signal format and count rate at the maximum operating speed. A serial feedback interface may provide high-resolution position data, but its update interval and latency must be appropriate for the servo architecture. A fine sensor value that arrives too slowly, or with excessive delay variation, may not improve dynamic control.

The measurement point determines which errors are visible

A motor-mounted rotary encoder measures motor rotation. It does not directly measure the position of a table driven through a screw, belt, gearbox, or other transmission. Backlash, screw pitch error, torsional windup, bearing movement, and structural deformation can all create a difference between motor position and load position.

A linear scale mounted to measure the moving axis can include more of those transmission errors in the feedback loop. That makes direct load-position correction possible, but it does not eliminate every mechanical error. The scale reads displacement along its measurement axis at its mounting location. Pitch, yaw, roll, and deformation between the scale and the point of use can still cause an error at the tool or sensor.

This distinction is especially important when the measurement axis is offset from the working point. A small angular error multiplied by that offset produces a linear position error, often described as an Abbe error. For example, an angular deviation of approximately 10 microradians across a 100 mm offset corresponds to about 1 µm of displacement for small angles. An encoder with nanometre-scale output increments cannot independently correct that geometric error unless the relevant motion is measured and compensated.

Dual-loop arrangements can use motor feedback for inner-loop control and linear-scale feedback for load position. Their effectiveness depends on tuning and mechanical behavior; flexible transmission elements can make the two measured positions respond differently. A load encoder should therefore be assessed as part of the complete control system, not treated as an automatic cure for compliance.

Thermal drift can consume the resolution advantage

A feedback device measures within a physical structure whose dimensions change with temperature. A useful first approximation for uniform linear expansion is:

Change in length = expansion coefficient × original length × temperature change.

For illustration, a 200 mm component with a linear expansion coefficient of 12 × 10⁻⁶/°C changes length by approximately 2.4 µm for a uniform 1 °C temperature change. The result is an example of sensitivity, not a prediction for a particular machine: material, temperature distribution, constraints, and the location of the scale all affect the actual error.

Motors, drives, bearings, ambient air, and the process itself can create nonuniform heating. A scale attached to one part of a structure may not track movement at a separately heated tool or workpiece. Even if the servo holds the scale reading constant, the process point can drift relative to its reference.

Compensation can reduce predictable thermal error, but it requires temperature measurements and a model that represent the relevant structure. Warm-up state and measurement duration should be included in any comparison of sub-micron positioning results. A short, controlled repeatability test does not establish long-duration stability.

Static position and motion-path performance are different tests

For an inspection axis, a critical requirement may be settling at a series of locations and remaining stable during image acquisition. Feedback noise, settling time, vibration, and drift during exposure can matter more than the smallest displayed increment. If the position signal is averaged over time, the reported stability must still be related to the exposure period and the physical motion of the optical system.

For laser processing or precision CNC, the relevant requirement may include following a path while moving. Feedback resolution contributes to position observability, but contour error also depends on servo bandwidth, following error, axis synchronization, mechanical dynamics, and the trajectory. A system can show good point-to-point repeatability yet produce an unacceptable path at speed.

Motion reversal deserves particular attention. Backlash, friction, and servo behavior can produce a different position error depending on the approach direction. A single-direction repeatability result may be useful for a process that always approaches from one side, but it does not demonstrate equivalent bidirectional performance.

Comparing specifications and test evidence

A credible comparison starts with a defined measurand: which point is being evaluated, along which axis, over what travel, and relative to what reference. The required result should also state whether it concerns absolute positioning, repeatability, short-term stability, path tracking, or a combination of these.

When reviewing a proposed system, ask for the conditions behind the numbers:

• Whether the stated resolution is the sensor’s native signal period, interpolated output increment, or controller display increment.
• The sensor’s accuracy and interpolation-error specifications, including the conditions and travel range to which they apply.
• Position noise at rest, with its statistical measure, bandwidth or sampling conditions, and observation time.
• Whether positioning results are unidirectional or bidirectional, and whether compensation was enabled.
• Payload, speed, acceleration, mounting arrangement, environmental conditions, and thermal state during the test.
• The location and uncertainty of the independent measurement reference used to assess the machine.

For machine tools, the ISO 230 series includes test-code methods relevant to geometric and positioning performance; ISO 230-2 addresses the determination of positioning accuracy and repeatability of numerically controlled axes. The applicable edition, test procedure, and reported parameters should be identified before comparing results. A reference to a standard alone does not establish that two tests used the same travel, conditions, or acceptance limits.

Where uncertainty is important, the measurement reference should be substantially better characterized than the error being evaluated. Calibration traceability, instrument uncertainty, alignment, environmental correction, and the relationship between the reference measurement line and the process point all affect the strength of the result. Compensation settings should be documented so that calibrated performance is not confused with uncorrected mechanical performance.

The most useful encoder resolution is therefore the one that supports the required measurement and control performance with adequate margin, not necessarily the smallest number on a datasheet. If noise, geometry, or thermal movement already exceeds the allowable error, adding counts alone does not remove the limiting mechanism. The decisive evidence is a system-level measurement made at the relevant point, under representative operating conditions.