A Jupyter widget for annotating images with bounding boxes
A Jupyter widget for annotating images with bounding boxes. Try a live demo on Binder.
from jupyter_bbox_widget import BBoxWidgetwidget = BBoxWidget(image='fruit.jpg',classes=['apple', 'orange', 'pear'],)widget
Create, edit, move, resize and delete bounding box annotations using the mouse.
Use widget.bboxes to get current annotations values:
widget.bboxes# [{'x': 377, 'y': 177, 'width': 181, 'height': 201, 'label': 'apple'},# {'x': 219, 'y': 142, 'width': 169, 'height': 171, 'label': 'orange'},# {'x': 43, 'y': 174, 'width': 234, 'height': 195, 'label': 'pear'}]
You can also assign to widget.bboxes to display any annotations. For example, use the output of an object detection model to do model-assisted labeling.
widget.bboxes = [{'x': 377, 'y': 177, 'width': 181, 'height': 201, 'label': 'apple'},{'x': 219, 'y': 142, 'width': 169, 'height': 171, 'label': 'orange'},{'x': 43, 'y': 174, 'width': 234, 'height': 195, 'label': 'pear'}]
Fruit photo by @umanoide?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Umanoide on Unsplash
You can install using pip:
pip install jupyter_bbox_widget
A Jupyter server restart may be necessary for the widget to be properly discovered.
When you click inside the widget area it will gain focus and start receiving keyboard events. An outline usually indicates that the element is focused. Normal Jupyter keyboard shortcuts won’t work in this state. To unfocus the widget area click outside it or press Esc.
Some shortcuts act on the selected bbox. New bboxes are selected automatically when created. You can also select a bbox by clicking on it. Selected bbox is displayed on top of others and with a thicker border.
Esc unfocuses the widgetEnter is the same as pressing Submit buttonSpace is the same as pressing Skip buttonTab / Shift-Tab select next/previous bbox.W move upA move leftS move downD move rightQ shrink widthE grow widthR grow heightF shrink heightC assign selected class labelShift while pressing movement keys will increase step sizeYou can define functions that will be called automatically when you press Skip or Submit buttons. This is useful for creating a workflow for annotating multiple images.
@widget.on_skipdef skip():# move on to the next image@widget.on_submitdef save():# do stuff to save current annotations and move on
There is an example of a simple annotation workflow in examples/introduction.ipynb notebook.
Get or set widget state by using these traits.
image - path to a local image file or a web linkbboxes - list of bounding boxesclasses - list of class labelscolors - list of colors to use for different classeslabel - currently selected class labelselected_index - index of currently selected bbox. Is -1 when nothing is selected.hide_buttons - default False, remove Skip and Submit buttonsview_only - default False, make bboxes non-editable. This is useful for viewing annotation outputs without accidentally changing them.image_bytes - binary data from the image fileIt’s also possible to react to state changes by using the widget’s observe method. For example, the following code will make the function on_bbox_change run every time the user edits bounding boxes:
def on_bbox_change(change):new_bboxes = change['new']# do whatever with themwidget.observe(on_bbox_change, names=['bboxes'])
Maybe your images aren’t files in a common format or require special handling to load. One way to show them is to save the image into an in-memory bytes buffer and feed that to widget.image_bytes. I’m open to suggestions on how to make this more user-friendly.
from io import BytesIObytes_io = BytesIO()# for example, say img is a PIL imageimg.save(bytes_io, format='png')# feed that data to the widgetwidget.image_bytes = bytes_io.getvalue()
Sometimes you need to record more info about an object than just a location and a class label. For example, you might want to specify whether the object is in focus or blurred, record its size or other properties.
Let’s say we want to apply a rating on a scale from 1 to 5 to every object in the image. We create a slider widget to edit the rating values:
w_rating = widgets.IntSlider(value=3, min=1, max=5, description='Rating')
And we attach it to the bbox widget.
widget.attach(w_rating, name='rating')
As a result all bboxes created afterwards will have a rating property and the w_rating widget can be used to display and manipulate the rating of the currently selected bbox.
Any number and any kind of ipywidgets widgets may be used in this way for creating richer annotations - number inputs, text inputs, checkboxes and so on (see widget list).
The notebook in examples/introduction.ipynb has an example and a more detailed explanation of this feature.
anywidget under the hoodwidget.on_skip and widget.on_submit methods as decoratorswidget.labelhide_buttons optionview_only modeThis project was originally inspired by a blogpost Creating Reactive Jupyter Widgets With Svelte but is currently based on anywidget which provides a very nice developer experience.
Follow the steps below to make changes to this widget.
pip install -e .[dev]npm install to install JS dependencies.npm run dev to launch vite dev server. It will watch for any changes you make to the files in svelte folder.src/jupyter_bbox_widget/bbox.py change the DEV variable to True so that the widget gets its JS code from the vite dev server.ANYWIDGET_HMR=1 jupyter lab to turn on anywidget’s hot module reloading.hatch build to build your widget when it’s ready.