RoboSAM: A robot that is smart enough to call humans for help!

In my opinion, one of the most important attributes of being smart is the ability to seek help when needed. This requires realizing that help is needed and getting the right kind of help from the right source. Currently, robots do not have an ability to assess whether they can successfully complete a task or not. When instructed to do a task, they simply attempt to do it. Sometimes the task execution results in spectacular success that delights the spectators and other times it leads to an embarrassing failure that baffles everyone, except the person who programmed the robot. Clearly, if robots were to become smart, they will need to ask for help when they are unable to do a task. 

Occasional robot failures can be tolerated. However, using humans to frequently clean up the mess created by robots is simply not a viable business model for using robots. Currently, deploying robots in industrial applications requires the reliability of robotic task execution to be very high. This is accomplished by designing specialized hardware and software. Extensive system testing is needed to ensure that potential failure modes are well understood and contingency plans are developed to handle them. Typically, task execution failures shut down the line and require human intervention to clear the fault and restart the line. This type of intervention is very expensive and hence robots are not used on a task until extremely high-level reliability can be achieved. Customized hardware and software costs can only be justified if the production volume is sufficiently high and tasks are repetitive (e.g., automotive assembly lines). 

To understand the underlying challenges in robot deployment, consider the following scenario. A robot is capable of picking a part if it is presented to the robot at a certain location. However, if the part has shifted from its nominal location, the robot might not be able to grasp it. The robot does not simply know where the transition boundary between task execution success and failure lies. If the part is sufficiently distant from its expected location, as the robot attempts to grasp it, the robot might bump into it, push it further, and jam the material handling system. This can in turn trigger a system fault and shut down the system. 

In order to use robots in small production batch operations or non-repetitive tasks, we will need robots that are able to estimate the probability of task completion before beginning the task. This will enable robots to assess their own confidence in doing a task. If the robot does not have high confidence in completing a task, then it should call for help. This will enable human operators to provide the robot with needed assistance (e.g., better part pose estimation, invoking a different grasping strategy) and prevent major system faults that result from task execution failure. Please keep in mind that the human only needs to help the robot with the portion of the task that is proving to be challenging. The robot can do the rest itself. In most situations, providing task assistance help to robots is much cheaper than recovering from a system shutdown. 

My students have been building a robot to demonstrate this concept in the bin picking context. This project is called RoboSAM (ROBOtic Smart Assistant for Manufacturing). Bin picking capability is representative of a robot’s ability to perceive the desired object in the environment and to successfully pick it up and deliver it in a known pose. If the robot is not sure whether it can pick the desired part from a bin containing many different parts, then it calls a remotely located human operator for help. We call this operational concept human-on-call concept. This is fundamentally different from the human-in-the-loop concept that requires the human operator to actively monitor the manufacturing cell and take control away from the robot when the robot is about to make a mistake. The new concept requires the robot to call the human operator when it decides that it needs help. 

I believe that human-on-the-call concept is the right economic model for deploying robots. It enables humans to move away from doing boring routine tasks to do challenging tasks with which robots struggle. This model allows a single remotely situated human operator to help multiple robots on an “as needed” basis. It also enables robots to be deployed on tasks on which achieving very high success rate will be difficult. For the near foreseeable future, a large number of tasks in small and medium manufacturing companies fall in this category. 

People often ask what humans will do when robots become more widespread. In my opinion, humans will be needed to teach robots how to do different tasks and bail robots out when they are confused. The key will be to develop technologies that allow robots to ask for help when needed. Recent work in our lab is a step in that direction.

Recent Advances in Industrial Robots and Their Implications on Manufacturing

Industrial robots (e.g., ABB, PUMA) have been quite successful in mass production assembly lines. For example, they are routinely used to weld, paint, and join parts in automobile industry. However, small and medium manufacturers (SMM) in the US have largely stayed away from using industrial robots. They continue to rely on manual labor and this makes it hard for them to compete with overseas suppliers with low labor costs.

The National Association of Manufacturers (NAM) defines small manufacturers as companies with 500 or fewer employees and medium-sized manufacturers as companies with 2,500 or fewer employees. The NAM estimates that that the US has close to 300,000 SMM, representing a very important segment of the manufacturing sector. As we move towards shorter product life cycles and customized products, the future of manufacturing in the US will depend upon the ability of SMM to remain cost competitive.

This blog post explores the reasons behind the lack of adoption of industrial robotics technology by SMM and recent advances in robotics that might change the status quo.

Let us explore a representative scenario to understand the limitations of the current industrial robots and why they are not used by SMM. Imagine that you are working in a small company and building a prototype of new medical device. You are under extreme time pressure to meet an important deadline. As you are assembling the device, you realize the bracket is too compliant. You need to laser cut it again in a much stiffer material. The good news is that it will only take six minutes to cut the bracket. But the logistics associated with it will take an hour. You really need to continue assembling the rest of the assembly and testing the controller. You simply don’t have an hour to spend and can certainly use an assistant right now!

Here is what you would like your assistant to do - walk over to the material storage area, locate the right material, pick up the material, take it to the laser cutter, open the laser cutter, place the material in it, press the button to start cutting, wait for the part to finish, open the laser cutter, pick up the part, clean it, and bring it to you. Obviously human assistants can do all of these tasks without even flexing their cognitive muscles. I am sure that they can do all of these tasks while texting and surfing the net on their smart phones! Unfortunately the current industrial robots simply cannot do these tasks. So you simply cannot get a robot assistant today!

Robots that rule the assembly line have the following four limitations. First, they are immobile. They cannot go to the task location. The work has to be brought to them. Second, their dexterity is extremely limited. Simple tasks such as opening shelves and precisely placing and securing a previously unseen part in a machine are out of their capabilities. Third, it takes a long time to program them. So using robots on no-repetitive tasks is simply counter-productive. Finally, robots cannot work in the close proximity of humans because of safety concerns. So you can forget about a robot assistant walking over and handing you a tool or a part to assist you on the shop floor.

Most SMM use highly automated machines (e.g., CNC machines, laser cutter, water-jet cutters, CNC press-brakes, 3D printers). However, SMM shop floors tend to be unstructured and often go through changes to meet the needs of the projects at hand. Main sources of manual labor in SMM are material transport and handling, machine setup and calibration, inspection, clean-up, and packaging. Unfortunately, the current industrial robots that are designed for mass production assembly lines are of not much use in these tasks. So industrial robots offer very little value to SMM!

Recent advances in robotics are challenging the status quo and aiming to turn robots into important tools for SMM. I would like to share the following important trends:

• Mobile manipulators are robots that can transport themselves to the work site. I recently saw demonstrations of mobile manipulators developed by Kuka that show impressive capabilities. This capability will be very useful in expanding the role of robots in manufacturing, particularly from the SMM point of view.

• Dexterity has been a major obstacle to the widespread use of robots in manufacturing. Recent developments on robot hands are targeting to overcome this obstacle (e.g., Schunk and Barrett hands). 3D printing enables users to quickly create their own customized grippers in few hours.

• Baxter from Rethink Robotics is aiming to eliminate the need for writing code to program robots. Instead, robots can be programmed by demonstrating the tasks. This is expected to empower workers on the shop floor. They will be able to start utilizing robots without the need to wait for a robot programmer to assist them.

• Recent advances in human-safe robots are enabling robots to work in the close proximity of humans. Kuka lightweight arm and Baxter are representative examples of advances in this area. Many researchers are developing methods to track human operators in the workspace to make robots aware of humans in the workspace and change planned robot motions to avert injury to humans. For example, +Krishnanand Kaipa , +Carlos Morato , and +Boxuan Zhao  in my lab have developed a system to monitor a human operator working in the close proximity of a robot using four Microsoft Kinect sensors. This information is used by the robot to update its plan. The video of this system is shown below.

I believe that ultimately the convergence of the above mentioned technologies will create the second generation of industrial robots that will revolutionize the manufacturing industry. 

Once the demand increases for these robots, the cost for them will start coming down. There is no reason why low-end industrial robots cannot be sold for less than ten thousand dollars once the economy of scale kicks in. This in turn will make robots affordable for SMM and manufacturing cost-competitive in high wage countries.