GPU Particles with custom rendering/updating shaders
Our current Particle system supports CPU particles. Add support for GPU particles to allow us to render/update more particles. Allow users to provide shaders for custom updating/rendering of these particles. In addition, add support for rendering meshes and other shapes instead of just billboarded quads. Some of the work from https://github.com/highfidelity/hifi/pull/8324 can be revived for this
Open Source Modkit
⓵ 𝙒𝙝𝙖𝙩: 𝘼 𝙁𝙧𝙖𝙢𝙚𝙬𝙤𝙧𝙠 𝙛𝙤𝙧 𝙈𝙤𝙙𝙙𝙞𝙣𝙜 We all want to see the High Fidelity platform evolve and grow… From an open source standpoint, however, it can be difficult to identify the best ways to experiment and contribute results. It would be great if development paths were made clearer, so that more people could extend the platform and maintain results in parallel. And in order to protect contributor investments, extensions would need compatibility with stock builds and better odds of working across releases. We therefore suggest creating a *completely open source-minded* “Modkit” to help foster such contributions alongside the main application code base. ⓶ 𝙃𝙤𝙬: 𝙃𝙞𝙁𝙞 𝙊𝙥𝙚𝙣 𝙎𝙤𝙪𝙧𝙘𝙚 𝙈𝙤𝙙𝙠𝙞𝙩 With High Fidelity’s assistance we would like to bootstrap the following project components: ○ https://github.com/highfidelity/hifi/tree/master/plugins/hifiModkit (proposed) — New built-in plugin ○ https://github.com/highfidelity/hifi-modkit-sdk (proposed) — New SDK and github repo ○ https://modkit.highfidelity.io (proposed) — New API/SDK docs An expanded version of the above is available given corresponding interest. ⓷ 𝙒𝙝𝙮: 𝙋𝙚𝙤𝙥𝙡𝙚 𝙒𝙖𝙣𝙩 𝙩𝙤 𝘾𝙤𝙣𝙩𝙧𝙞𝙗𝙪𝙩𝙚 Lots of people have imagined and several have attempted extension of the High Fidelity platform, only to promptly hit significant roadblocks. And while the existing codebase has a “plugin” concept, it is neither well-documented nor well-suited for *ongoing* development. 𝗖𝘂𝗿𝗿𝗲𝗻𝘁𝗹𝘆❟ 𝘁𝗵𝗲 𝗿𝗲𝗾𝘂𝗶𝗿𝗲𝗱 𝗲𝗳𝗳𝗼𝗿𝘁 𝗼𝗳 𝗱𝗲𝘃𝗲𝗹𝗼𝗽𝗲𝗿𝘀 𝗶𝘀: ○ Conventional HiFi Plugin: VisualStudio, VisualStudio Dependencies, Qt, vcpkg, CMake, generate & regenerate CMake, Python, GBs of Disk Space, lib/plugins, lib/shared, Hours of Time 𝗪𝗶𝘁𝗵 𝗠𝗼𝗱𝗸𝗶𝘁 𝗶𝘁 𝗯𝗲𝗰𝗼𝗺𝗲𝘀: ○ Modkit Plugin: C compiler + hifi-modkit-sdk․zip Paving the way for future enhancements that live longer and remain effective will require a stronger foundation; a foundation like Modkit, which prioritizes open source collaboration and support for externally-managed extensions. 𝗪𝗔𝗡𝗧𝗘𝗗: [ Valve Index Controller support | Space Mouse support | Avatar Shaders | Custom Initialization / Default Scripts | Business Productivity Tools | Domain Access controls | External Chat Integration | Permissions management | Alternative Editing systems | Expanded JS APIs | Streaming Audio integrations | Advanced NPCs | Inventory Systems | Internet-of-Things (IoT) bridges | Additional Multimedia codecs | … ]
Bring back the built-in Script Editor...
Earlier versions of Interface.exe included a built-in script editor that enabled you to: * Create, load, and work on simple Client scripts without having to leave Interface. * Focus on, edit, and restart one script without fumbling through Running Scripts... or having to Reload all content. * See relevant console.* and print log messages for the focused script in a dedicated console panel. * Interactively try one-liners and inspect/modify global variables live while the focused script is running. A few years ago this feature was removed by HiFi for apparent reason of being too "Desktop only." It was so useful to me, however, that I have maintained a working version as part of my local builds. There has been no great way to share this with others because it requires forking, patching, and compiling your own Interface.exe from source. I propose to package this up and contribute it instead as an optional and externally-maintained application extension under the Modkit framework. DEPENDS ON MODKIT
Qt WebView integration for Quest and Android
Because, QTWebEngine is not available on mobile platforms, where policy dictates that all web content is displayed using the operating system’s web view, can be used instead, Qt WebView, wich provide a way to display web content in a QML application without necessarily including a full web browser stack, by using native API’s where it make sense. This implementation will alllow web browsing on Android and Quest platforms. https://doc.qt.io/qt-5/qml-qtwebview-webview.html https://doc.qt.io/qt-5/qtwebview-minibrowser-example.html https://code.qt.io/cgit/qt/qtwebview.git/tree/examples/webview/minibrowser?h=5.13
Speech recognition STT integration
Would be great to have a speech to text integrated in High Fidelity, for NPC, language learning, transcription of presentations, translation, etc. For this purpose we can use some free, open source toolkits like CMUSphinx: https://cmusphinx.github.io/2015/10/qtspeechrecognition-api-for-qt-using-pocketsphinx/ https://cmusphinx.github.io/ http://jrmeyer.github.io/asr/2016/01/09/Installing-CMU-Sphinx-on-Ubuntu.html or Julius: https://github.com/julius-speech/julius
Upgrade the "DerpQ" state of the art RL AI integration into HiFi
The plan is to bring the type of AI into High Fidelity that Unity is working on here https://unity3d.com/machine-learning I already have a proof of concept implimentation at these links and described in the video. https://youtu.be/UcXe2HqY0sQ https://gym.openai.com/ https://github.com/neureal/gym-zmq https://github.com/wilbown/hifi/tree/wilbown https://github.com/openai/baselines/tree/master/baselines/deepq
Procedural Vertex Shaders
Related to http://roadmap.highfidelity.com/open-source-project-proposals/p/procedural-materials-on-models-and-avatars Allow Procedurals to specify a custom vertex shader. If an entity has a procedural vertex shader, it should get its bounding box from the material entity. Part of this work was done here, but it needs to be redone and allow the material entity to specify the render bounds: https://github.com/highfidelity/hifi/pull/13823
I'd like to open a discussion on how to implement inventory. After surveying a few other virtual worlds, the concept of user-owned inventory seems to be missing. Also, the marketplace seems to be vestigial in HF at the moment (please correct me if I'm wrong), and I think the concept of owning digital content should not be overlooked. After some thought, I really still like the blockchain certificate concept from HF, but it would be nice if it could be pinned to something already in existence and in use. I would really like to see people able to buy from each other 1-to-1 and shop for and purchase online the way we purchase anything else, via paypal or some other merchant checkout. For the concept of people running their own servers, it probably isn't too far a stretch to expect people to carry their own inventory as well. How to allow this and maintain creator protection is something I'd like to talk about more and put some ideas on the table.
Audio monitor demo
Audio Client Interface with just Spatial Audio. One can then have audio-only conversation in-world. A simple use case would be a "baby monitor" with just an audio connection. I'm doing this as one of project tutorials to on-board new developers.
Make the Create app work nicely with inspect.js and derivatives
When using the inspect.js script with the Create app active you are likely to unexpectedly duplicate entities. This is because the Create app re-uses the Alt key to duplicate entities. Option A: Change the Create app to use Ctrl+mouse-down to duplicate entities (instead of Alt+mouse-down), and not do anything for Alt+mouse-down Option B: Change the Create app to use Shift+mouse-down to duplicate entities (instead of Alt+mouse-down), and the Ctrl key to toggle the display the “railroad” axes (instead of the Shift key), and not do anything for Alt+mouse-down. Option C: Add a menu option to the Create app’s menu items, “Duplicate entities with Ctrl instead of Alt”, and make inspect.js also read the menu setting and use the key that Create isn’t using. (I.e., toggle between current behaviour and option A.) Background The inspect.js script uses the Alt key plus Ctrl and Shift combinations to orbit and pan the camera about any entity or avatar – the same control scheme as Second Life. (This control scheme is likely to be second nature to people who have come from Second Life.) - Alt + mouse up/down: zooms in / out. - Alt + mouse left/right: orbits left / right. - Alt + Ctrl + mouse up/down/left/right: orbits over / under / left / right. - Alt + Ctrl + Shift + mouse up/down/left/right: pans down / up / right / left. Unfortunately, the Create app uses the Alt key to duplicate entities. While the “official” inspect.js isn’t so useful for positioning the camera while using the Create app because the view snaps back to first person when the Alt key is released, 3rd party versions of inspect.js leave the camera where it is which is much more convenient for editing. (In the 3rd party versions, to snap back to first person, the user can press Esc or move their avatar.) Which raises another option to perhaps do in addition to one of the above … Option D: Update inspect.js with some of the improvements that 3rd parties have made to the script. In particular, the Fox Essentials (Cutelab) version has collected together various improvements from a number of people, including the CtrlAltStudio improvements.