{ "cells": [ { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "%matplotlib inline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n# Simulating Argus II\n\n## Background\n\n:py:class:`~pulse2percept.implants.ArgusII` is an epiretinal implant developed\nby Second Sight Medical Products, Inc. (Sylmar, CA).\nNow discontinued, it was the first retinal implant to get FDA approval in the US\nand the CE mark in Europe, and has been implanted in close to 500 patients\nworldwide.\n\nA number of studies have documented how the artificial vision provided by\n:py:class:`~pulse2percept.implants.ArgusII` (Second Sight Medical Products, Inc.)\ndiffers from normal sight.\nArgus II contains 60 electrodes of 225 um diameter arranged in a 6 x 10\ngrid (575 um center-to-center separation) [Yue2020]_.\nSome researchers therefore assumed that the stimulation of a grid of\nelectrodes on the retina would lead to the perception of a grid of luminous\ndots (\"phosphenes\").\nWe refer to this as the :py:class:`~pulse2percept.models.ScoreboardModel`\nof prosthetic vision.\nHowever, a growing body of evidence has shown that retinal implant users often\nreport seeing distorted phosphenes and require extensive rehabilitative training \nto make use of their new vision [Beyeler2019]_, [EricksonDavis2021]_.\n\nAlthough single-electrode phosphenes are consistent from trial to trial, they \nvary across electrodes and users [Luo2016]_, [Beyeler2019]_.\nPhosphene shape strongly depends on stimulation parameters [Nanduri2012]_\nas well as the retinal location of the stimulating electrode [Beyeler2019]_ \ndue to inadvertent activation of passing axon fibers in the retina.\nThe result is a rich repertoire of phosphene shape that includes blobs, arcs,\nwedges, and triangles [Beyeler2019]_:\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "import matplotlib.pyplot as plt\nimport pulse2percept as p2p\n\nfig, axes = plt.subplots(ncols=3, figsize=(10, 3))\nfor ax, subject, scale in zip(axes, ['S2', 'S3', 'S4'], [1, 1, 0.5]):\n data = p2p.datasets.fetch_beyeler2019(subjects=subject)\n p2p.viz.plot_argus_phosphenes(data, ax=ax, scale=scale)\n ax.axis('off')\n ax.set_title(subject)\nfig.tight_layout()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "These phosphene shapes can be simulated with the\n:py:class:`~pulse2percept.models.AxonMapModel`, which was developed to fit\nbehavioral data (see [Beyeler2019]_ for details).\n\n## Boston Train sequence\n\nTo simulate the vision provided by Argus II, we first need to set up a new\naxon map model. We can specify phosphene size (``rho``) and elongation\n(``axlambda``) as well as the visual field we would like to simulate (given\nin degrees of visual angle):\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "model = p2p.models.AxonMapModel(rho=400, axlambda=200,\n xrange=(-12, 12), yrange=(-8, 8))\nmodel.build()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can visualize where the implant sits on the axon map as follows:\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "model.plot()\np2p.implants.ArgusII().plot()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We then need to choose a stimulus to run through the model:\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "p2p.stimuli.BostonTrain().play()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In real life, the Argus II camera would capture a video just like the above,\nconvert it to grayscale, downscale it, and then assign each pixel to an\nelectrode in the implant.\n\nAfter feeding the resulting stimulus through the axon map model, we get a\npretty good idea of what this video would look like to an Argus II patient:\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "implant = p2p.implants.ArgusII(stim=p2p.stimuli.BostonTrain())\nmodel.predict_percept(implant).play()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The above simulation is basically equivalent to the scoreboard model, because\neach electrode appears as a large blob.\n\nHowever, as mentioned above, every patient sees phosphenes differently.\nTo some they appear thin and elongated, to others they appear big and\narc-like.\n\nTo increase the arc length of individual phosphenes, we can choose a larger\n``axlambda`` value:\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "model.axlambda = 600\nmodel.build()\nmodel.predict_percept(implant).play()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "On the other hand, some retinal implant recipients (e.g., 51-009) report\nseeing thin and elongated phosphenes. In this case, the same video may appear\nvery different to them:\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "model.rho = 50\nmodel.axlambda = 1000\nmodel.build()\nmodel.predict_percept(implant).play()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Girl Pool sequence\n\nAnother video shows a girl jumping into a swimming pool:\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "p2p.stimuli.GirlPool().play()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Similar to the above video, we can convert it to grayscale and downscale it,\nthen feed it through the axon map model:\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "implant = p2p.implants.ArgusII(stim=p2p.stimuli.GirlPool())\nmodel.rho = 400\nmodel.axlambda = 200\nmodel.build()\nmodel.predict_percept(implant).play()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here is the same video with longer phosphenes:\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "model.axlambda = 600\nmodel.build()\nmodel.predict_percept(implant).play()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here is the same video with thin and long phosphenes:\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "model.rho = 50\nmodel.axlambda = 1000\nmodel.build()\nmodel.predict_percept(implant).play()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In reality, the vision provided by Argus II may be even worse for several\nreasons:\n\n1. The above simulations do not consider temporal distortions (e.g., flicker,\n persistence, fading)\n2. For some patients, individual phosphenes do not assemble into more complex\n percepts, or do so in a highly unpredictable way.\n\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.9" } }, "nbformat": 4, "nbformat_minor": 0 }