Examples: Vision
This page shows how to use Ned’s Vision Set.
Note
Even if you do not own a Vision Set, you can still realize these examples with the Gazebo simulation version.
Danger
If you are using the real robot, make sure the environment around it is clear.
Needed piece of code
Important
In order to achieve the following examples, you need to
create a vision workspace. In this page, the workspace used is named workspace_1
.
To create it, the user should go on Niryo Studio!
As the examples start always the same, add the following lines at the beginning of codes:
# Imports
from pyniryo import *
# - Constants
workspace_name = "workspace_1" # Robot's Workspace Name
robot_ip_address = "x.x.x.x"
# The pose from where the image processing happens
observation_pose = PoseObject(
x=0.16, y=0.0, z=0.35,
roll=0.0, pitch=1.57, yaw=0.0,
)
# Place pose
place_pose = PoseObject(
x=0.0, y=-0.2, z=0.12,
roll=0.0, pitch=1.57, yaw=-1.57
)
# - Initialization
# Connect to robot
robot = NiryoRobot(robot_ip_address)
# Calibrate robot if the robot needs calibration
robot.calibrate_auto()
# Updating tool
robot.update_tool()
# --- -------------- --- #
# --- CODE GOES HERE --- #
# --- -------------- --- #
robot.close_connection()
Hint
All the following examples are only a part of what can be made with the API in terms of Vision. We advise you to look at API - Vision to understand more deeply
Simple Vision Pick & Place
The goal of a Vision Pick & Place is the same as a classical Pick & Place, with a close difference: the camera detects where the robot has to go in order to pick!
This short example shows how to do your first Vision pick using the
vision_pick()
function:
robot.move_pose(observation_pose)
# Trying to pick target using camera
obj_found, shape_ret, color_ret = robot.vision_pick(workspace_name)
if obj_found:
robot.place_from_pose(place_pose)
robot.set_learning_mode(True)
Code Details - Simple Vision Pick and Place
To execute a Vision pick, we firstly need to go to a place where the robot will be able to see the workspace:
robot.move_pose(observation_pose)
Then, we try to perform a Vision pick in the workspace with the
vision_pick()
function:
obj_found, shape_ret, color_ret = robot.vision_pick(workspace_name)
Variables shape_ret
and color_ret
are respectively of type
ObjectShape
and ObjectColor
, and
store the shape and the color of the detected object! We will not use them for this first
example.
The obj_found
variable is a boolean which indicates whereas an
object has been found and picked, or not. Thus, if the pick worked,
we can place the object at the place pose.
if obj_found:
robot.place_from_pose(place_pose)
Finally, we turn learning mode on:
robot.set_learning_mode(True)
Note
If your obj_found
variable indicates False
, check that:
Nothing obstructs the camera field of view
Workspace’s 4 markers are visible
At least 1 object is placed fully inside the workspace
First conditioning via Vision
In most of use cases, the robot will need to perform more than one Pick & Place. In this example, we will see how to condition multiple objects according to a straight line:
# Initializing variables
offset_size = 0.05
max_catch_count = 4
# Loop until enough objects have been caught
catch_count = 0
while catch_count < max_catch_count:
# Moving to observation pose
robot.move_pose(observation_pose)
# Trying to get object via Vision Pick
obj_found, shape, color = robot.vision_pick(workspace_name)
if not obj_found:
robot.wait(0.1)
continue
# Calculate place pose and going to place the object
next_place_pose = place_pose.copy_with_offsets(x_offset=catch_count * offset_size)
robot.place_from_pose(next_place_pose)
catch_count += 1
robot.go_to_sleep()
Code Details - First Conditioning via Vision
We want to catch max_catch_count
objects, and space each of
them by offset_size
meter:
offset_size = 0.05
max_catch_count = 4
We start a loop until the robot has caught max_catch_count
objects:
catch_count = 0
while catch_count < max_catch_count:
For each iteration, we firstly go to the observation pose and then, try to make a Vision pick in the workspace:
robot.move_pose(observation_pose)
obj_found, shape, color = robot.vision_pick(workspace_name)
If the Vision pick failed, we wait 0.1 second and then, start a new iteration:
if not obj_found:
robot.wait(0.1)
continue
Else, we compute the new place position according to the number of catches, and then, go placing the object at that place:
next_place_pose = place_pose.copy_with_offsets(x_offset=catch_count * offset_size)
robot.place_from_pose(next_place_pose)
We also increment the catch_count
variable:
catch_count += 1
Once the target catch number is achieved, we go to sleep:
robot.go_to_sleep()
Multi Reference Conditioning
During a conditioning task, objects may not always be placed as the same
place according to their type. In this example, we will see how to align object
according to their color, using the
color element ObjectColor
returned by vision_pick()
function:
# Distance between elements
offset_size = 0.05
max_failure_count = 3
# Dict to write catch history
count_dict = {
ObjectColor.BLUE: 0,
ObjectColor.RED: 0,
ObjectColor.GREEN: 0,
}
try_without_success = 0
# Loop until too much failures
while try_without_success < max_failure_count:
# Moving to observation pose
robot.move_pose(observation_pose)
# Trying to get object via Vision Pick
obj_found, shape, color = robot.vision_pick(workspace_name)
if not obj_found:
try_without_success += 1
robot.wait(0.1)
continue
# Choose X position according to how the color line is filled
offset_x_ind = count_dict[color]
# Choose Y position according to ObjectColor
if color == ObjectColor.BLUE:
offset_y_ind = -1
elif color == ObjectColor.RED:
offset_y_ind = 0
else:
offset_y_ind = 1
# Going to place the object
next_place_pose = place_pose.copy_with_offsets(x_offset=offset_x_ind * offset_size,
y_offset=offset_y_ind * offset_size)
robot.place_from_pose(next_place_pose)
# Increment count
count_dict[color] += 1
try_without_success = 0
robot.go_to_sleep()
Code Details - Multi Reference Conditioning
We want to catch objects until Vision Pick failed max_failure_count
times.
Each of the object will be put on a specific column according to its color.
The number of catches for each color will be stored on a dictionary count_dict
.
# Distance between elements
offset_size = 0.05
max_failure_count = 3
# Dict to write catch history
count_dict = {
ObjectColor.BLUE: 0,
ObjectColor.RED: 0,
ObjectColor.GREEN: 0,
}
try_without_success = 0
# Loop until too much failures
while try_without_success < max_failure_count:
For each iteration, we firstly go to the observation pose and then, try to make a Vision pick in the workspace:
robot.move_pose(observation_pose)
obj_found, shape, color = robot.vision_pick(workspace_name)
If the Vision pick failed, we wait 0.1 second and then, start a new iteration, without forgetting to increment the failure counter:
if not obj_found:
try_without_success += 1
robot.wait(0.1)
continue
Else, we compute the new place position according to the number of catches, and then, go place the object at that place:
# Choose X position according to how the color line is filled
offset_x_ind = count_dict[color]
# Choose Y position according to ObjectColor
if color == ObjectColor.BLUE:
offset_y_ind = -1
elif color == ObjectColor.RED:
offset_y_ind = 0
else:
offset_y_ind = 1
# Going to place the object
next_place_pose = place_pose.copy_with_offsets(x_offset=offset_x_ind * offset_size,
y_offset=offset_y_ind * offset_size)
robot.place_from_pose(next_place_pose)
We increment the count_dict
dictionary and reset try_without_success
:
count_dict[color] += 1
try_without_success = 0
Once the target catch number is achieved, we go to sleep:
robot.go_to_sleep()
Sorting Pick with Conveyor
An interesting way to bring objects to the robot, is the use of a Conveyor Belt. In this examples, we will see how to catch only a certain type of object by stopping the conveyor as soon as the object is detected on the workspace.
# Initializing variables
offset_size = 0.05
max_catch_count = 4
shape_expected = ObjectShape.CIRCLE
color_expected = ObjectColor.RED
conveyor_id = robot.set_conveyor()
catch_count = 0
while catch_count < max_catch_count:
# Turning conveyor on
robot.run_conveyor(conveyor_id)
# Moving to observation pose
robot.move_pose(observation_pose)
# Check if object is in the workspace
obj_found, pos_array, shape, color = robot.detect_object(workspace_name,
shape=shape_expected,
color=color_expected)
if not obj_found:
robot.wait(0.5) # Wait to let the conveyor turn a bit
continue
# Stopping conveyor
robot.stop_conveyor(conveyor_id)
# Making a vision pick
obj_found, shape, color = robot.vision_pick(workspace_name,
shape=shape_expected,
color=color_expected)
if not obj_found: # If visual pick did not work
continue
# Calculate place pose and going to place the object
next_place_pose = place_pose.copy_with_offsets(x_offset=catch_count * offset_size)
robot.place_from_pose(next_place_pose)
catch_count += 1
# Stopping & unsetting conveyor
robot.stop_conveyor(conveyor_id)
robot.unset_conveyor(conveyor_id)
robot.go_to_sleep()
Code Details - Sort Picking
Firstly, we initialize your process: we want the robot to catch 4 red circles. To do so,
we set variables shape_expected
and color_expected
with
ObjectShape.CIRCLE
and ObjectColor.RED
.
offset_size = 0.05
max_catch_count = 4
shape_expected = ObjectShape.CIRCLE
color_expected = ObjectColor.RED
We activate the connection with the Conveyor Belt and
start a loop until the robot has caught max_catch_count
objects:
conveyor_id = robot.set_conveyor()
catch_count = 0
while catch_count < max_catch_count:
For each iteration, we firstly run the Conveyor Belt (if the latter is already running, nothing will happen), then go to the observation pose:
# Turning the Conveyor Belt on
robot.run_conveyor(conveyor_id)
# Moving to observation pose
robot.move_pose(observation_pose)
We then check if an object corresponding to our criteria is in the workspace. If not, we wait 0.5 second and then, start a new iteration:
obj_found, pos_array, shape, color = robot.detect_object(workspace_name,
shape=shape_expected,
color=color_expected)
if not obj_found:
robot.wait(0.5) # Wait to let the conveyor turn a bit
continue
Else, stop the Conveyor Belt and try to make a Vision pick:
# Stopping Conveyor Belt
robot.stop_conveyor(conveyor_id)
# Making a Vision pick
obj_found, shape, color = robot.vision_pick(workspace_name,
shape=shape_expected,
color=color_expected)
if not obj_found: # If visual pick did not work
continue
If Vision Pick succeed, compute new place pose, and place the object:
# Calculate place pose and going to place the object
next_place_pose = place_pose.copy_with_offsets(x_offset=catch_count * offset_size)
robot.place_from_pose(next_place_pose)
catch_count += 1
Once the target catch number is achieved, we stop the Conveyor Belt and go to sleep:
# Stopping & unsetting Conveyor Belt
robot.stop_conveyor(conveyor_id)
robot.unset_conveyor(conveyor_id)
robot.go_to_sleep()