Robotics Software Engineer Interview Questions: What to Ask and What to Look For

Q: "You are designing a communication layer between a perception node running at 30Hz and a control node running at 1kHz. How do you handle the rate mismatch?"

Strong answer: Recognizes this as a fundamental robotics systems problem. The control node cannot block waiting for perception data. Discusses several approaches: the perception node publishes at 30Hz and the control node reads the latest available data (lock-free single-producer single-consumer pattern), with interpolation or extrapolation between perception updates to provide smooth input at 1kHz. Explains timestamping: every perception message needs a hardware timestamp so the control node can account for the age of the data. Discusses thread safety: how to ensure the control thread is never blocked by the perception thread (lock-free data structures, double-buffering, or atomic pointer swaps). Mentions zero-copy transport options for large data (point clouds, images) to avoid unnecessary copies. May discuss what happens when perception drops frames and how the control node degrades gracefully.

Red flags: Ignores the timing problem entirely. Suggests the control node blocks on perception data. Does not mention thread safety or lock-free patterns. Has no strategy for handling dropped frames or late data.

Q: "How do you design a software system that can be tested without physical hardware?"

Strong answer: Describes a layered abstraction approach. Hardware interfaces are abstracted behind well-defined APIs so that the same application code can run against real hardware, a physics simulator (Gazebo, Isaac Sim, MuJoCo), or lightweight mock implementations. Discusses the testing pyramid for robotics: unit tests for algorithms and logic (fast, no hardware), integration tests against simulated hardware (medium speed, test component interactions), and hardware-in-the-loop tests for final validation (slow, require physical systems). Explains how CI/CD pipelines use simulation to catch regressions before code reaches hardware. Mentions the importance of recording real hardware data (rosbags) and replaying it in tests to validate perception and planning changes against known ground truth.

Red flags: Says "we test on the robot." Does not understand hardware abstraction layers. Cannot describe how to run automated tests without hardware. Has no concept of a testing pyramid for robotics.

Q: "Your robot software stack has a memory leak that causes a crash after 3 hours of operation. How do you find it?"

Strong answer: Describes a systematic investigation. First, narrows down which process is leaking by monitoring process memory over time (top, htop, or custom metrics). Then uses profiling tools to identify the allocation site: Valgrind with the Massif heap profiler for detailed analysis, AddressSanitizer (ASan) with leak detection enabled for faster feedback, or custom memory tracking instrumentation. Discusses common robotics-specific memory leak patterns: accumulating sensor data in buffers that are not bounded, ROS message queues growing unbounded, cached transform data that is never pruned, or GPU memory leaks from perception pipeline. Explains how to reproduce reliably: if the crash takes 3 hours, accelerate it by running with smaller buffers or higher data rates. Mentions that in C++, tools like Valgrind and ASan are invaluable, while in Python, tracemalloc and objgraph help track down reference cycles and growing containers.

Red flags: Suggests restarting the robot every 2 hours as a workaround. Does not know memory profiling tools. Cannot describe common leak patterns in robotics software. Has never debugged a long-running process.

Q: "Compare single-threaded event loops with multi-threaded architectures for robotics software. When would you use each?"

Strong answer: Understands the tradeoffs deeply. Single-threaded event loops (like the ROS2 single-threaded executor) provide deterministic execution order, no race conditions, and simpler debugging, but they cannot take advantage of multiple cores and a slow callback blocks everything. Multi-threaded architectures (ROS2 multi-threaded executor, custom thread pools) enable parallelism and prevent slow tasks from blocking fast ones, but introduce synchronization complexity, potential priority inversion, and harder-to-reproduce bugs. Recommends single-threaded for safety-critical control loops where determinism matters most, and multi-threaded for perception pipelines where parallelism provides real throughput benefits. May discuss the ROS2 callback group model as a middle ground, or the approach of running multiple single-threaded executors on pinned cores for both determinism and parallelism.

Red flags: Always uses one approach without considering the alternative. Does not understand the determinism implications. Cannot discuss race conditions or synchronization. Has never thought about CPU core assignment for robotics workloads.

Q: "How do you handle graceful shutdown of a robotics system with multiple interconnected nodes?"

Strong answer: Treats shutdown as a safety-critical operation. Describes a dependency-ordered shutdown sequence: first transition the robot to a safe state (stop motion, engage brakes, lower any lifted loads), then shut down application-level nodes in reverse dependency order, then shut down hardware interface nodes, and finally release hardware resources. Discusses ROS2 lifecycle management for coordinated state transitions across nodes. Explains what happens when a node does not shut down cleanly: watchdog timers that force-terminate after a deadline, and hardware safety systems (e-stop, mechanical brakes) that engage independently of software. Mentions that every node must handle SIGTERM gracefully and complete critical operations (like saving state or flushing logs) before exiting.

Red flags: Sends SIGKILL to everything. Does not consider robot safety during shutdown. Has no concept of ordered shutdown or lifecycle management. Does not handle the case where a node hangs during shutdown.

Q: "You need to deploy a software update to 100 robots in the field. How do you design the update system?"

Strong answer: Describes a robust OTA (over-the-air) update architecture. Key elements include: A/B partitions so the robot can roll back to the previous version if the update fails, staged rollout (deploy to 5% of the fleet, monitor for 24 hours, then expand), health checks that run automatically after an update to verify the robot is operating correctly, and automatic rollback if health checks fail. Discusses the update payload: full image updates for consistency vs differential updates for bandwidth efficiency. Mentions security considerations: signed update packages, verified boot chain, encrypted transport. Addresses the practical challenges: robots with intermittent connectivity, updates interrupted by power loss, and coordinating updates with robot operational schedules (do not update during a shift). May discuss container-based deployments (Docker/Podman) for easier rollback and version management.

Red flags: Pushes updates to all robots simultaneously. No rollback strategy. Does not consider what happens when an update fails mid-installation. No health monitoring after deployment. Ignores connectivity challenges for field robots.

Q: "How do you ensure your robotics software meets real-time constraints? What tools and techniques do you use?"

Strong answer: Describes a comprehensive approach from kernel to application. At the OS level: uses PREEMPT_RT patched kernel, isolates CPUs for real-time threads with isolcpus, disables CPU frequency scaling, and configures appropriate scheduling policies (SCHED_FIFO or SCHED_DEADLINE). At the application level: avoids dynamic memory allocation in the real-time path (pre-allocates everything), avoids system calls that can block (no file I/O, no logging to disk in the hot path), uses lock-free data structures for communication with non-real-time threads, and avoids page faults by locking memory with mlockall. For measurement and validation: uses cyclictest to characterize system latency, instruments the control loop to measure and log execution time, sets up alerts when latency exceeds thresholds, and runs stress tests (stress-ng) concurrently to verify worst-case behavior.

Red flags: Has never measured actual latency on a system. Does not know PREEMPT_RT or CPU isolation. Cannot explain what to avoid in a real-time thread. Thinks "use C++ instead of Python" is a sufficient answer for real-time. Has no measurement or validation methodology.

Q: "The controls team wants to run their node at 1kHz with guaranteed scheduling. The perception team wants to use all available GPU memory. How do you mediate?"

Strong answer: Approaches it as a system-level resource allocation problem, not a conflict. Quantifies the requirements: what exactly does the controls team need (CPU cores, memory bandwidth, scheduling latency) and what does the perception team need (GPU memory, GPU compute, CPU for pre/post processing). Proposes technical solutions: CPU isolation gives the controls team dedicated cores with guaranteed scheduling; GPU memory budgets prevent the perception team from starving other GPU consumers; separate process groups with cgroup limits prevent cross-team resource interference. Explains the decision framework: safety-critical workloads (controls) get resource guarantees first, then perception gets the remaining capacity. Facilitates agreement by defining an API contract between the teams rather than letting resource usage be implicit.

Red flags: Picks a side without understanding both requirements. Does not propose concrete technical mechanisms for resource isolation. Cannot think at the system level. Creates a political negotiation instead of a technical solution.

Q: "A customer reports intermittent failures that you cannot reproduce. How do you approach this?"

Strong answer: Starts by gathering information systematically. Asks the customer for specific details: when does it happen, what is the robot doing, how often, any pattern (time of day, specific task, after a certain duration). Reviews available telemetry and logs from the robot during the failure windows. If the on-robot logging is insufficient, deploys enhanced diagnostics to the affected robot(s) with the customer's permission. Formulates hypotheses based on the failure pattern: intermittent failures often point to race conditions, environmental factors (lighting, temperature, WiFi interference), or hardware degradation (loose connector, overheating component). Designs targeted tests to confirm or eliminate each hypothesis. Communicates progress to the customer regularly, even when the root cause is not yet found. Once identified, adds automated detection for the failure mode so it is caught proactively on other robots.

Red flags: Closes the ticket because it cannot be reproduced. Does not ask for details from the customer. Has no strategy for investigating intermittent issues. Does not consider environmental or hardware factors. No follow-through on preventing recurrence.

Q: "How do you onboard a new engineer to a large robotics codebase? What documentation matters most?"

Strong answer: Prioritizes the documentation that provides the fastest path to productivity. The most valuable document is an architecture overview that explains the system at a high level: what the major components are, how data flows, and where to find things. The second most valuable is a "getting started" guide that gets the new engineer from zero to running the full stack in simulation within a day. The third is a set of well-documented "starter tasks" that give the new engineer a meaningful contribution within the first week. Discusses the importance of code review as a learning tool: assigning the new engineer to review PRs from experienced engineers, not just having their code reviewed. Mentions pair programming or shadowing sessions for hardware-specific knowledge that is hard to document. Acknowledges that the best documentation is maintained alongside the code, not in a separate wiki that goes stale.

Red flags: Says "just read the code." Has never thought about onboarding. Relies entirely on tribal knowledge. Creates massive documentation dumps that no one reads. Does not mention hands-on experience with the robot as part of onboarding.

Q: "You notice a senior colleague is building a component that you think has a significant architectural flaw. How do you handle it?"

Strong answer: Approaches with curiosity first: schedules a conversation to understand the design rationale, because the senior engineer may have context or constraints they are not aware of. If after understanding the reasoning they still believe there is a flaw, presents their concern with evidence: a concrete scenario where the design would fail, a quantitative analysis showing a bottleneck, or precedent from a previous system. Proposes an alternative and is willing to build a prototype to demonstrate the tradeoff. Escalates through the design review process if there is no agreement. Accepts the outcome gracefully if the team decides to proceed with the original design, while documenting their concern for future reference.

Red flags: Stays silent because the colleague is senior. Publicly criticizes without first understanding the rationale. Escalates immediately to management. Cannot present their concern with concrete evidence. Takes it personally if the team disagrees.