Examples: Vision

This page shows how to use Ned’s vision set

If you want see more about Ned’s vision functions, you can look at PyNiryo - Vision

If you want to see how to do image processing, go check out the Pyniryo (first version) doc.

Note

Even if you do not own a Vision Set, you can still realize these examples with the Gazebo simulation version

Danger

If you are using the real robot, make sure the environment around it is clear

Needed piece of code

Important

In order to achieve following examples, you need to have create a vision workspace. In this page, the workspace used is named workspace_1. To create it, the user should go on Niryo Studio !

As the examples start always the same, add the following lines at the beginning of codes

# Imports
from pyniryo import *

# - Constants
workspace_name = "workspace_1"  # Robot's Workspace Name
robot_ip_address = "x.x.x.x"

# The pose from where the image processing happens
observation_pose = PoseObject(
    x=0.16, y=0.0, z=0.35,
    roll=0.0, pitch=1.57, yaw=0.0,
)
# Place pose
place_pose = PoseObject(
    x=0.0, y=-0.2, z=0.12,
    roll=0.0, pitch=1.57, yaw=-1.57
)

# - Initialization

# Connect to robot
robot = NiryoRobot(robot_ip_address)
# Calibrate robot if robot needs calibration
robot.arm.calibrate_auto()
# Updating tool
robot.tool.update_tool()

# --- -------------- --- #
# --- CODE GOES HERE --- #
# --- -------------- --- #

robot.end()

Hint

All the following examples are only of part of what can be made with the API in terms of vision. We advise you to look at API - Vision to understand more deeply

Simple Vision Pick & Place

The goal of a Vision Pick & Place is the same as a classical Pick & Place, with a close difference : the camera detects where the robot has to go in order to pick !

This short example show how to do your first vision pick using the vision_pick() function :

robot.move_pose(observation_pose)
# Trying to pick target using camera
obj_found, shape_ret, color_ret = robot.vision.vision_pick(workspace_name)
if obj_found:
    robot.pick_place.place_from_pose(place_pose)

robot.arm.set_learning_mode(True)

Code Details - Simple Vision Pick and Place

To execute a Vision pick, we firstly need to go to a place where the robot will be able to see the workspace

robot.arm.move_pose(observation_pose)

Then, we try to perform a vision pick in the workspace with the vision_pick() function

obj_found, shape_ret, color_ret = robot.vision.vision_pick(workspace_name)

Variables shape_ret and color_ret are respectively of type ObjectShape and ObjectColor, and store the shape and the color of the detected object ! We won’t use them for this first example.

The obj_found variable is a boolean which indicates whereas an object has been found and picked, or not. Thus, if the pick worked, we can go place the object at the place pose.

if obj_found:
    robot.pick_place.place_from_pose(place_pose)

Finally, we turn learning mode on:

robot.arm.set_learning_mode(True)

Note

If you obj_found variable indicates False, check that :

• Nothing obstruct the camera field of view

• Workspace’s 4 markers are visible

• At least 1 object is placed fully inside the workspace

First conditioning via Vision

In most of use cases, the robot will need to perform more than one Pick & Place. In this example, we will see how to condition multiple objects according to a straight line

# Initializing variables
offset_size = 0.05
max_catch_count = 4

# Loop until enough objects have been caught
catch_count = 0
while catch_count < max_catch_count:
    # Moving to observation pose
    robot.arm.move_pose(observation_pose)

    # Trying to get object via Vision Pick
    obj_found, shape, color = robot.vision.vision_pick(workspace_name)
    if not obj_found:
        robot.wait(0.1)
        continue

    # Calculate place pose and going to place the object
    next_place_pose = place_pose.copy_with_offsets(x_offset=catch_count * offset_size)
    robot.pick_place.place_from_pose(next_place_pose)

    catch_count += 1

robot.arm.go_to_sleep()

Code Details - First Conditioning via Vision

We want to catch max_catch_count objects, and space each of them by offset_size meter

offset_size = 0.05
max_catch_count = 4

We start a loop until the robot has caught max_catch_count objects

catch_count = 0
while catch_count < max_catch_count:

For each iteration, we firstly go to the observation pose and then, try to make a vision pick in the workspace

robot.arm.move_pose(observation_pose)

obj_found, shape, color = robot.vision.vision_pick(workspace_name)

If the vision pick failed, we wait 0.1 second and then, start a new iteration

if not obj_found:
    robot.wait(0.1)
    continue

Else, we compute the new place position according to the number of catches, and then, go placing the object at that place

next_place_pose = place_pose.copy_with_offsets(x_offset=catch_count * offset_size)
robot.pick_place.place_from_pose(next_place_pose)

We also increment the catch_count variable

catch_count += 1

Once the target catch number is achieved, we go to sleep

robot.arm.go_to_sleep()

Multi Reference Conditioning

During a conditioning task, objects may not always be placed as the same place according to their type. In this example, we will see how to align object according to their color, using the color element ObjectColor returned by vision_pick() function

# Distance between elements
offset_size = 0.05
max_failure_count = 3

# Dict to write catch history
count_dict = {
    ObjectColor.BLUE: 0,
    ObjectColor.RED: 0,
    ObjectColor.GREEN: 0,
}

try_without_success = 0
# Loop until too much failures
while try_without_success < max_failure_count:
    # Moving to observation pose
    robot.arm.move_pose(observation_pose)
    # Trying to get object via Vision Pick
    obj_found, shape, color = robot.vision.vision_pick(workspace_name)
    if not obj_found:
        try_without_success += 1
        robot.wait(0.1)
        continue

    # Choose X position according to how the color line is filled
    offset_x_ind = count_dict[color]

    # Choose Y position according to ObjectColor
    if color == ObjectColor.BLUE:
        offset_y_ind = -1
    elif color == ObjectColor.RED:
        offset_y_ind = 0
    else:
        offset_y_ind = 1

    # Going to place the object
    next_place_pose = place_pose.copy_with_offsets(x_offset=offset_x_ind * offset_size,
                                                   y_offset=offset_y_ind * offset_size)
    robot.pick_place.place_from_pose(next_place_pose)

    # Increment count
    count_dict[color] += 1
    try_without_success = 0

robot.arm.go_to_sleep()

Code Details - Multi Reference Conditioning

We want to catch objects until Vision Pick failed max_failure_count times. Each of the object will be put on a specific column according to its color. The number of catches for each color will be store on a dictionary count_dict

# Distance between elements
offset_size = 0.05
max_failure_count = 3

# Dict to write catch history
count_dict = {
    ObjectColor.BLUE: 0,
    ObjectColor.RED: 0,
    ObjectColor.GREEN: 0,
}

try_without_success = 0
# Loop until too much failures
while try_without_success < max_failure_count:

For each iteration, we firstly go to the observation pose and then, try to make a vision pick in the workspace

robot.move_pose(observation_pose)

obj_found, shape, color = robot.vision.vision_pick(workspace_name)

If the vision pick failed, we wait 0.1 second and then, start a new iteration, without forgetting the increment the failure counter

if not obj_found:
    try_without_success += 1
    robot.wait(0.1)
    continue

Else, we compute the new place position according to the number of catches, and then, go placing the object at that place

# Choose X position according to how the color line is filled
offset_x_ind = count_dict[color]

# Choose Y position according to ObjectColor
if color == ObjectColor.BLUE:
    offset_y_ind = -1
elif color == ObjectColor.RED:
    offset_y_ind = 0
else:
    offset_y_ind = 1

# Going to place the object
next_place_pose = place_pose.copy_with_offsets(x_offset=offset_x_ind * offset_size,
                                               y_offset=offset_y_ind * offset_size)
robot.pick_place.place_from_pose(next_place_pose)

We increment the count_dict dictionary and reset try_without_success

count_dict[color] += 1
try_without_success = 0

Once the target catch number is achieved, we go to sleep

robot.arm.go_to_sleep()

Sorting Pick with Conveyor

An interesting way to bring objects to the robot, is the use of a Conveyor Belt. In this examples, we will see how to catch only a certain type of object by stopping the conveyor as soon as the object is detected on the workspace

# Initializing variables
offset_size = 0.05
max_catch_count = 4
shape_expected = ObjectShape.CIRCLE
color_expected = ObjectColor.RED

conveyor_id = robot.conveyor.set_conveyor()

catch_count = 0
while catch_count < max_catch_count:
    # Turning conveyor on
    robot.conveyor.run_conveyor(conveyor_id)
    # Moving to observation pose
    robot.arm.move_pose(observation_pose)
    # Check if object is in the workspace
    obj_found, pos_array, shape, color = robot.vision.detect_object(workspace_name,
                                                                    shape=shape_expected,
                                                                    color=color_expected)
    if not obj_found:
        robot.wait(0.5)  # Wait to let the conveyor turn a bit
        continue
    # Stopping conveyor
    robot.conveyor.stop_conveyor(conveyor_id)
    # Making a vision pick
    obj_found, shape, color = robot.visionvision_pick(workspace_name,
                                                      shape=shape_expected,
                                                      color=color_expected)
    if not obj_found:  # If visual pick did not work
        continue

    # Calculate place pose and going to place the object
    next_place_pose = place_pose.copy_with_offsets(x_offset=catch_count * offset_size)
    robot.pick_place.place_from_pose(next_place_pose)

    catch_count += 1

# Stopping & unsetting conveyor
robot.conveyor.stop_conveyor(conveyor_id)
robot.conveyor.unset_conveyor(conveyor_id)

robot.arm.go_to_sleep()

Code Details - Sort Picking

Firstly, we initialize your process : we want the robot to catch 4 Red Circles. To do so, we set variables shape_expected and color_expected with ObjectShape.CIRCLE and ObjectColor.RED

offset_size = 0.05
max_catch_count = 4
shape_expected = ObjectShape.CIRCLE
color_expected = ObjectColor.RED

We activate the connection with the conveyor and start a loop until the robot has caught max_catch_count objects

conveyor_id = robot.set_conveyor()

catch_count = 0
while catch_count < max_catch_count:

For each iteration, we firstly run the conveyor belt (if the later is already running, nothing will happen), then go to the observation pose

# Turning conveyor on
robot.conveyor.run_conveyor(conveyor_id)
# Moving to observation pose
robot.arm.move_pose(observation_pose)

We then check if an object corresponding to our criteria is in the workspace. If not, we wait 0.5 second and then, start a new iteration

obj_found, pos_array, shape, color = robot.vision.detect_object(workspace_name,
                                                                shape=shape_expected,
                                                                color=color_expected)
if not obj_found:
    robot.wait(0.5)  # Wait to let the conveyor turn a bit
    continue

Else, stop the conveyor and try to make a vision pick

# Stopping conveyor
robot.conveyor.stop_conveyor(conveyor_id)
# Making a vision pick
obj_found, shape, color = robot.vision.vision_pick(workspace_name,
                                                   shape=shape_expected,
                                                   color=color_expected)
if not obj_found:  # If visual pick did not work
    continue

If Vision Pick succeed, compute new place pose, and place the object

# Calculate place pose and going to place the object
next_place_pose = place_pose.copy_with_offsets(x_offset=catch_count * offset_size)
robot.pick_place.place_from_pose(next_place_pose)

catch_count += 1

Once the target catch number is achieved, we stop the conveyor and go to sleep

# Stopping & unsetting conveyor
robot.conveyor.stop_conveyor(conveyor_id)
robot.conveyor.unset_conveyor(conveyor_id)

robot.arm.go_to_sleep()