{ "cells": [ { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "%matplotlib inline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n# Generating a stimulus from a video\n\n*This example shows how to use videos as input stimuli for a retinal implant.*\n\n## Loading a video\n\nA video can be loaded as follows:\n\n.. code:: python\n\n stim = p2p.stimuli.videos.VideoStimulus(\"path-to-video.mp4\")\n\nThere is an example video that is pre-installed with pulse2percept. You can\nload it like this.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "import pulse2percept as p2p\nimport numpy as np\n\nvideo = p2p.stimuli.BostonTrain(as_gray=True)\nprint(video)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There is a lot of useful information in this output.\n\nFirstly, note that ``vid_shape`` gives the dimension of the original video in\n(height, width, the number of frames).\n\nOn the other hand, ``shape`` gives the dimension of the stimulation which is\n(the number of electrodes, the number of time steps). This is calculated from\nflattening the video to (height x width, the number of frames).\n\nThe video data is stored as a 2D NumPy array in ``video.data``. You can\nreshape it to the original video dimensions as follows:\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "data = video.data.reshape(video.vid_shape)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Then it's possibly to access individual pixels or frames by indexing into the\nNumPy array. For example, to plot the first frame of the movie, use:\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "import matplotlib.pyplot as plt\nplt.imshow(data[..., 0], cmap='gray')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Preprocessing a video\n\nA :py:class:`~pulse2percept.stimuli.VideoStimulus` object comes with a number\nof methods to process a video before it is passed to an implant. Some\nexamples include:\n\n- :py:meth:`~pulse2percept.stimuli.VideoStimulus.invert` the gray levels of\n the video,\n- :py:meth:`~pulse2percept.stimuli.VideoStimulus.resize` the video,\n- :py:meth:`~pulse2percept.stimuli.VideoStimulus.rotate` the video,\n- :py:meth:`~pulse2percept.stimuli.VideoStimulus.filter` each frame of the\n video and extract edges (e.g., Sobel, Scharr, Canny, median filter),\n- :py:meth:`~pulse2percept.stimuli.VideoStimulus.apply` any input-output\n function not provided: The function (applied to each frame of the video)\n must accept a 2D or 3D image and return an image with the same dimensions.\n\nFor a complete list, check out the documentation for\n:py:class:`~pulse2percept.stimuli.VideoStimulus`.\n\nLet's do some processing for our example video. Firstly, let's play the video\nso that we know what it looks like originally.\n(It might take a couple of seconds for the video to appear, because it needs\nto be converted to HTML & JavaScript code first.)\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "video.play()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For example, let's resize the video to 100 x 100, and then use the Sobel\nfilter to extract edges. This can be done in one line. Then we will play\nthe processed video:\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "edge_video = video.resize((100, 100)).filter('Sobel')\nedge_video.play()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As demonstrated above, multiple video processing steps can be performed in\none line. This is possible because each method returns a copy of the\nprocessed video (without altering the original).\nThis means you can combine as many different processing steps as you like:\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "video.resize((40, 40)).rotate(10).invert().filter('median').play()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Using the video as input to a retinal implant\n\n:py:class:`~pulse2percept.stimuli.VideoStimulus` can be used in\ncombination with any :py:meth:`~pulse2percept.implants.ProsthesisSystem`.\nTo make this work, the stimulus needs to have the same number of pixels as\nthe implant has electrodes.\n\nIn most cases, the implant contains a rectangular grid of electrodes, so\nwe just have to resize the video first so that the number of pixels in each\nframe of the video matches the number of electrodes in the implant:\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "implant = p2p.implants.ArgusII()\n# Assign the resized video to the implant:\nimplant.stim = video.resize(implant.shape)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Then you can feed the video directly into any of the available models\ndescribed by a :py:class:`~pulse2percept.models.Model` object, such as the\naxon map model:\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "model = p2p.models.AxonMapModel()\nmodel.build()\npercept = model.predict_percept(implant)\npercept.play()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Lastly, we can save the percept to disk:\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "percept.save('video_percept.mp4')" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.9" } }, "nbformat": 4, "nbformat_minor": 0 }