Dao Will Go Visual!

[daokoder]

Please understand that, I am not yet ready to disclose more information.
😎

[dumblob]

Nice to see you back @daokoder!

The visualization looks promising - I'm curious what is going on 😉.

Speaking about visualization, in the last half a year, the library https://github.com/vurtun/zahnrad became in my eyes superseeding the ImGui library in basically all architectural and technical ways (but unfortunately not yet in features). Might be worth taking a look at.

[Night-walker]

Got me 😲 :)

[daokoder]

Speaking about visualization, in the last half a year, the library https://github.com/vurtun/zahnrad became in my eyes superseeding the ImGui library in basically all architectural and technical ways (but unfortunately not yet in features). Might be worth taking a look at.

It looks interesting, and might be exactly what I have been looking for :)

Got me 😲 :)

Appears so 😃

[daokoder]

While developing the visual version of Dao, I have found some unnecessary complexities underlying the Dao implementation. These complexities were introduced apparently for some seemingly unessential features. I will take this opportunity of developing the visual version to reevaluate the trade-off between functionalities and complexities of the features, and try to reduce the underlying complexity by removing or reducing some features. Such simplification will also reduce the gap between the vanilla version and the visual version of Dao.

The followings have been considered to be removed:

• Syntax macro support:

It will be easier to support DSL in the visual version, so there
should be no need for the syntax macro module and a few tweaks
inside the Dao parser and VM;

• Code inlining for mixed language programming:

Again it will be easier to support it in the visual version;

• Routine decorator and class decorator:

Mainly to simplify things (such as call argument expansion etc.);

• Call argument expansion ( call(x, ...) ):

It is not very complicated, but adds a lot implementation burdens;

Simplifications might be made to:

• Constant folding:

Change non-trivial evaluation from compiling time to start-up time;

More will be updated here when more come to my mind.

[dumblob]

Well, the listed changes seem to me too drastic (especially when the reasoning is that supporting them in the visual Dao variant is difficult).

For example I totally love evaluation of as much as possible in compile time, because it saves extreme amount of time and nerves (imagine that you need to run a whole deployment of an application having half a milion SLOC just to test if somewhere in the middle of the application running time some constant folding does what it should).

Call argument expansion is a very handy feature (I'm using it quite regularly and can't imagine programming in Dao without them) - it makes the language way more versatile (versatility is e.g. one of the killer features of Python).

Routine and Class decorators are essential for many high-level concepts in strongly typed languages like Dao. Again, I can't imagine my life without them in Dao. It would make me to copy paste code and type more etc. It's a significant step back 😢.

Code inlining for mixed language programming - I haven't used it yet (because I wasn't using Dao on mixed projects using different languages), but from the reactions of audience on the talk I had about Dao I can tell, that it's a really very good advertising feature.

Syntax macro support - I haven't used it for any serious stuff yet, but I liked the idea (e.g. #376). In Go, there has had not been any macros nor any directives for the compiler, but over the time the pressure and need for something like that increased extremely and nowadays Go supports quite a few advanced compiler directives in comments. So the need exists for sure, but if it could be substituted for something else, then let's try it.

So I'm not so much convinced that we should remove these features just because it's complex. It is, but yet more useful. Or am I missing something?

Actually I'm perceiving Dao as kind of stable from the features point of view, so even because of this I'm far from being keen on removing the mentioned features.

[daokoder]

Well, the listed changes seem to me too drastic (especially when the reasoning is that supporting them in the visual Dao variant is difficult).

Only removing decorator may seem drastic, the others not really.

For example I totally love evaluation of as much as possible in compile time, because it saves extreme amount of time and nerves (imagine that you need to run a whole deployment of an application having half a milion SLOC just to test if somewhere in the middle of the application running time some constant folding does what it should).

Maybe I was not very clear when I say "start-up time". Actually in terms of execution time, what I have in mind is essentially the same as before. It will just move constant folding from the parsers to a separated routine (could be called __init__()), so that constant folding for a single script file can be executed at once. So it will actually become more efficient in this regard.

Also, this way it will also simplify the parsing slightly and the bytecode generation/loading significantly. This change will require adding a few VM instructions such as DVM_LOAD, DVM_SETCL, DVM_SETCK and DVM_SETCG, so the expressing of a compiled script in bytecode will become more complete and self-contained, making it easier to target to the DaoVM from other languages.

Call argument expansion is a very handy feature (I'm using it quite regularly and can't imagine programming in Dao without them) - it makes the language way more versatile (versatility is e.g. one of the killer features of Python).

I know it is handy for users, I just doubt is is very useful. Maybe there are use scenarios I am not familiar with, so I will not push to remove it. The same goes for decorators.

Code inlining for mixed language programming - I haven't used it yet (because I wasn't using Dao on mixed projects using different languages), but from the reactions of audience on the talk I had about Dao I can tell, that it's a really very good advertising feature.

Syntax macro support - I haven't used it for any serious stuff yet, but

I am still strongly inclined to remove them. They are completely redundant in the visual variant. And there way better ways to support them in the visual variant. It is too ugly to support them in the vanilla version.

[dumblob]

Ok, thank you for the thorough explanation. Initially I misunderstood some of your proposals. I fully agree with all of the changes you've right now described with the small exception of cell argument expansion and decorators.

There is no special use case known to me, where cell argument expansion or decorators could not be substituted, but just the fact they simplify and clarify the written code seems good enough reason for trying to preserve them.

@Night-walker, do you have any other insights?

[dumblob]

Offtopic: @daokoder, where are you maintaining your code right now when the domain daovm.net apparently serves for other purposes? I would like to ressurrect the domain and in the mean time use a similar one instead.

I'm willing to buy VPS (or more) in central Europe for you, because I'm highly interested in maintaining Dao further for real-life applications (I've prepared few projects and therefore I would very much like to pay the group of people who will work on Dao and with Dao through https://salt.bountysource.com/ ).

[dumblob]

Any news on this @daokoder ?

[daokoder]

Any news on this @daokoder ?

Not much on the visual language per se, besides the improvements to Dao and the VM as mentioned in issue #527. But there has been some good progress on integrating Dao and the visual programming environment with the Atomic Game Engine. (This engine will be used to develop plugins for robot, game and app development, to make the environment suitable for learning programming, robotics and science. In time, it will also become suitable for software development, robot development and data analysis etc. by professionals.)

[dumblob]

In the meantime Zahnrad (mentioned above) became Nuklear, but what's more important, the author created another GUI library "gui.c" (see also other single-header stuff on https://gist.github.com/vurtun ), which is something like a golden grail among anything regarding 2D UI. It's significantly smaller than Nuklear, but features actually "everything" (because of it's "total" modularity - a well-managed set of kinda independent rectangle-canvases representing every single part of the UI).

It's based on the idea, that one needs all three "modes" of UI specification in reality (because Nuklear is only "immediate"):

1. pure declarative (i.e. template-based/retain-mode)
2. semi immediate (the templates are constructed in run-time - in the best case just at the initialization/startup of the app, in the worst case each frame; the rest is handled the same as in the pure declarative mode)
3. pure immediate (i.e. code-based; everything is handled immediately/just_in_time)

I'm writing it here, because I'm literally amazed. I can finally say it somehow realizes my cca 10 years of ideas in the UI domain of IT (pretty much like Dao realizes my 6 years of ideas in the programming languages domain of IT 😉).

[dumblob]

Just curious how it goes. Any news? We've decided to use Dao for a semi large scale project with the hope that we'll get some development resources also for Dao itself in the upcoming year.

@daokoder are you using Dao also in a bigger-than-small production environments? Any hints?

[daokoder]

Just curious how it goes. Any news? We've decided to use Dao for a semi large scale project with the hope that we'll get some development resources also for Dao itself in the upcoming year.

@daokoder are you using Dao also in a bigger-than-small production environments? Any hints?

Nice to hear from you. Currently I am developing a game inspired by Minecraft, Civilization and a couple of other games. It was initially prototyped in Dao. Though now it is mostly written in C++, but Dao programming interface is still supported and reserved for modding in the future.

The development of the visual programming is suspended for the time being. It has become more or less feature complete, but still missing many minor things, and will require a significant amount of time to become release-ready. I will resume on it when the game starts to demand less time (likely the next year), and integrate it with the game when it becomes ready.

In the mean time, my colleague is continuing on working a virtual robot environment we started a couple of years ago. It is mostly developed in Dao on top of a C++ game engine.

So yeah, we are actively using Dao in production environments, and will continue to do so in the foreseeable future:)

[dumblob]

I'm glad you're fine. Thanks for the reply.

Nice to hear from you. Currently I am developing a game inspired by Minecraft, Civilization and a couple of other games. It was initially prototyped in Dao. Though now it is mostly written in C++, but Dao programming interface is still supported and reserved for modding in the future.

Well, this is quite interesting. What were the reasons to switch from Dao to C++? I would expect it to happen the other way round.

[daokoder]

Well, this is quite interesting. What were the reasons to switch from Dao to C++? I would expect it to happen the other way round.

I probably have given the wrong impression, but it was actually not much a switching. The initial prototyping was every basic, meant to test a couple of very basic ideas just out of curiosity. It was done in just a couple of weeks, then I decided to do more serious development using C++. The choice of C++ was primary derived from the fact that I needed to adapt a couple of C++ classes of the game engine to make it more suitable for the game I intended to develop.

Dao was still used for input and UI for some time. I tried to bring in a friend to assist me with this part. But it didn't turn out well, partially due to inadequate experience in his part, and partially due to insufficient documentation and API exposure in the game engine part. The game engine's UI component classes are wrapped around a third-party UI library, they only provide a partially wrapping of the classes and methods of the library. The Dao wrappers were generated from the engine classes, though they contained more or less a complete wrapper for the UI component, it provided only a partially coverage of the functionalities of the underlying UI library.

So when I decided to do the UI part myself, I found out that it was actually much easier to build UI in C++ than in Dao, due to direct access to the underlying UI library. And it took me less than two weeks to finish what my friend had done in months. At that point, I decide to only use C++ for developing the game (another reason for this is that it should become harder for players to hack the game). Still Dao and a couple of its modules are still used internally in the game as C libraries.

[dumblob]

I tried to bring in a friend to assist me with this part. But it didn't turn out well, partially due to inadequate experience in his part, and partially due to insufficient documentation and API exposure in the game engine part.

I'm sorry to hear that.

Either way, I'd like to support the game you're making. Once you'll release it, I'll buy it even though I don't play any games as I want to support Dao in the real world.

Btw in the semi large scale project we'll need to run some code in a web browser, but for the time being only ECMAScript (or its "enhanced" counterparts like TypeScript) is available. I would prefer to write this code in some interpreted language with sane semantics and not ECMAScript, but currently only C/C++/Go compiles to WebAssembly in a usable way. Lua is on its way, but it'll take a year or two to get there (see https://github.com/appcypher/awesome-wasm-langs ) - GC specification in WebAssembly is not yet ready (but MVP was recently released, yay!). Dao VM compiled to JS is too slow and big for our needs.

So I think we'll anyway need to choose JS for this small part of the project. Or would you have any other recommendation?

[daokoder]

Either way, I'd like to support the game you're making. Once you'll release it, I'll buy it even though I don't play any games as I want to support Dao in the real world.

Thanks. I have been considering to list it in the Dao website as an option for people to support Dao.

So I think we'll anyway need to choose JS for this small part of the project. Or would you have any other recommendation?

If it is just a small part of the project, JS should be an acceptable option. Currently any other option will likely require a lot more work.

[dumblob]

By the way which build system (or IDE) are you using for the game development to be productive? We are considering "continuous compilation" offered e.g. by https://github.com/sandstorm-io/ekam .

[daokoder]

By the way which build system (or IDE) are you using for the game development to be productive?

I use qmake + cmake (for the game engine) + daomake (for Dao). The reason for using qmake was that the game was initially developed as a plugin for the visual programming IDE which is developed using Qt.

[dumblob]

@daokoder how is your game and the visual Dao going? Haven't heard about you for a while and your web site states 2018 together with Visual programming language, virtual robot and creative game. (Not yet ready to release). (daoscript.org says even 2016).

[daokoder]

@daokoder how is your game and the visual Dao going? Haven't heard about you for a while and your web site states 2018 together with Visual programming language, virtual robot and creative game. (Not yet ready to release). (daoscript.org says even 2016).

@dumblob
It has been a very busy period of time, as I was trying to finish the game before the end of this Chinese year. Now the Chinese New Year is just a couple of days away, it turns out that I was a bit overly optimistic about the development. Though the development was actually quite smooth and fast, but there are just too many things to be implemented. Fortunately, all of the important parts and most of the basic parts are done, I should be able to finish the remaining things in a couple of months, hopefully it will become ready for release in later March or April.

As for the visual programming language (I actually have decided a new name for it, but let's just call it visual Dao for now), I haven't found the time for it yet, hopefully I will manage to spend some time on it after the game is released and improved with a few major updates. Anyway, I will try to resume the development of visual Dao as soon as possible after the game release, and do my best to release it this year. Actually, one of my goals is to integrate the IDE for visual Dao with a tailed version of the game, that should become something very interesting ^_^

PS. The footnotes in daoscript.org webpages indeed need some updates.

[dumblob]

@daokoder any updates? The game, the "Visual Dao", the footnotes, any ideas on the newer issues in this GitHub Dao issues tracker?

Over the time, there is one thing which I lack more and more in Dao - support for higher level "streams" ("flows", ... you name it) - i.e.:

1. streams which are infinite (i.e. do not have any end like e.g. file streams or string streams and thus also do not have any length),

2. which support signalization of being "overwhelmed" (preferably implicitly e.g. like anything based on Hoares CSP does),

3. which are not homogeneous (i.e. you can change the streamed data type any time - basically supporting Dao type any),

4. which support fully variable frequency of streaming depending solely on both producers and consumers (this goes hand in hand with the backpropagation),

5. which support merging and splitting (including multiplexing and demultiplexing),

6. which run parallel (it's enough if just pseudo-parallel like in CSP, but with automated scheduling of all these pseudo-parallel threads on many/all CPU cores in a cluster automatically),

7. and which support runtime addition/deletion of producers and consumers.

The ultimate solution to this problem would probably not be a library, but some language-level ("first class citizens") constructs or in the worst case a fork of Dao with a shifted behavior to become a dataflow language - i.e. by default every statement, assignment and call (basically "every line of code") would run completely in parallel (Erlang virtual machine an its bytecode would be a good inspiration) and just those few requiring sequential processing would need to be enclosed e.g. in a code block - such fork would though need to retain all the awesomness of Dao 😉 (i.e. a modern, quite safe and very high level language though very easy to learn).

There is ParaSail which supports exactly this pervasive parallelism and I actually quite like the language (judging based on the bundled examples 😉), but it's not (yet?) so awesome as Dao (though, ParaSail authors seem to be quite open, so we can suggest implementing the Dao awesomness there 😉).

Quite recently I've also started following activities around "REBOL reincarnation" called Red (and its system programming language counterpart Red/System which the Red compiler itself is written in and which is basically another syntax for a subset of C). Some nice properties of Red might interest you - e.g. the reactive interface (a close counterpart of "streams" as outlined above, but this time achieved using actors). We shouldn't though compare Red to Dao as Red uses homoiconicity everywhere (all APIs are based on this feature and thus static analysis of Red is not easily possible).

[daokoder]

The main update I can give now is about the game. Though it has been long overdue, it is finally very close to finish. It has taken a few months longer than I planned, mainly because I added a few new things such as village system and various improvements along the way. Now only a few things remain to be finished or improved before the first release, and I am confident I can finish these things within this month. A public announcement of the game will come out within this month, and if nothing goes wrong, it will be released in later July or early August.

The game will support Dao as the primary language for modding. As soon as the game is ready for testing, I will spend some time to add to Dao a few improvements that have been hanging over my head for quite some time. If I will have more spare time, I will also spend some time on the development of "Visual Dao".

As for dataflow like language features, I am more in line with the idea of supporting dataflow based programming as Dao modules, and adding language level support in the "Visual Dao". The last thing I did on visual programming before switching to game development was a set of syntax macros to implement language level support for Arduino programming. It was a proof of concept implementation, but it did work nicely on small examples. My plan was to add full support for Arduino programming, I believe it is very feasible. I think similar thing can be done for dataflow based programming, with or without relevant modules for Dao.

[dumblob]

A public announcement of the game will come out within this month, and if nothing goes wrong, it will be released in later July or early August.

If you need alpha/beta testing, feel free to contact me (I'll be happy to sign NDA as I would like to support your work). Btw I have the opportunity to reach out to many thousands of people working with unix and Linux systems who are eager to see games running on these "non-Windows" PCs. If your game will run under MacOS and/or BSD and/or Linux, just tell me when and I'll make the public announcement (but be prepared for the initial load of visitors during the first 1-4 days which this will cause).

...few improvements that have been hanging over my head for quite some time...

Sounds very promising. Looking forward to that.

The last thing I did on visual programming before switching to game development was a set of syntax macros to implement language level support for Arduino programming. It was a proof of concept implementation, but it did work nicely on small examples.

This would interest me as I thought syntax macros were removed from Dao some time ago (back then it didn't seem that much of an issue, but now after seeing and understanding the steep rise of compile-time evaluation - Rust, D, Nim, ... - I would actually like to see them back).

[daokoder]

If you need alpha/beta testing, feel free to contact me (I'll be happy to sign NDA as I would like to support your work).

Thanks, I will let you know when it's ready for testing.

Btw I have the opportunity to reach out to many thousands of people working with unix and Linux systems who are eager to see games running on these "non-Windows" PCs. If your game will run under MacOS and/or BSD and/or Linux, just tell me when and I'll make the public announcement (but be prepared for the initial load of visitors during the first 1-4 days which this will cause).

I was considering to only support Mac and Windows initially, based on Steam user statistics. Now I will see if it is convenient to add Linux to the initially supported platforms.

This would interest me as I thought syntax macros were removed from Dao some time ago (back then it didn't seem that much of an issue, but now after seeing and understanding the steep rise of compile-time evaluation - Rust, D, Nim, ... - I would actually like to see them back).

Actually these syntax macros I was talking about are part of "Visual Dao". Syntax macros in "Visual Dao" are much more flexible and powerful than those removed from Dao. I made the decision to remove the old syntax macro support from Dao, because I was quite optimistic about the development of "Visual Dao" and felt it would become redundant once "Visual Dao" is release. But then I had to postpone the "Visual Dao" project after I switch to the game project. If you do actually need them in Dao, I can consider to bring them back, and wait until the release of "Visual Dao" to decide whether to keep or remove them.

[dumblob]

Actually these syntax macros I was talking about are part of "Visual Dao". Syntax macros in "Visual Dao" are much more flexible and powerful than those removed from Dao.

Now I understand. Then I'm eager to see them 😄.

If you do actually need them in Dao, I can consider to bring them back, and wait until the release of "Visual Dao" to decide whether to keep or remove them.

There is always a way around (though not so nice as macros usually add to the compile time safety checking), so don't bring them back if you're planning to release Visual Dao as open source (any ETA?).

[daokoder]

There is always a way around (though not so nice as macros usually add to the compile time safety checking), so don't bring them back if you're planning to release Visual Dao as open source (any ETA?).

Yes, it will be open sourced, at least the backend part that is required for compiling and running "Visual Dao" programs. The IDE part will also be open sourced eventually, but only after I have explored its potential to some extents.

Even for the backend part, I prefer to release its source some time after its binary release. It might be unwise to release the source too soon given the status of Dao adoption. If Dao and "Visual Dao" can build up a sizable community and ecosystem faster, I will be more comfortable to open source it sooner.

I am hoping the game with modding interface in Dao would be the first step to make this really happening. Then there is "Visual Dao" itself and its support for Arduino programming. There is also a game engine integration with the IDE to support game development using Dao. If done properly, these four should be able to jump-start a significant adoption of Dao and "Visual Dao".

any ETA?

I cannot say for sure now. It will depend on the game, if it is successful, I may be able to carry out my plans without interruption (even with speedup). Otherwise I might have to postpone it and resume it a couple/few of years later, or find a way to get one or all projects funded in some way (but the options I can see will surely complicate things).

[dumblob]

Just one more note to the "higher level" streams as I mentioned above. I kind of like the API design of "reactive engine" (which could actually be the number one use case for the streams I've outlined) implemented in Red - see https://doc.red-lang.org/en/reactivity.html#_concept . The Red's reactive engine supports both compile-time as well as run-time addition and removal of producers and consumers.

In Dao the easiest would probably be to implement "reactors" using async classes, but they were removed from Dao (though they're still part of documentation). So now the implementation would use channels and parallel functions to achieve similar results.

In Dao I'm looking forward to the opportunity of choosing which reactors shall run in a parallel thread (using !!) and which reactors shall stay as Hoares CSPs in the same thread. I would also like the "reactive" stream interface to have pluggable "transport layer" (by default memory sharing among threads using POSIX locking mechanisms - see e.g. Dask) and pluggable scheduler (i.e. opportunity to implement own heuristic how to schedule workers to threads and to physical machines across the network in case the transport layer will account not just for the current system, but also for remote machines - see again e.g. Dask).

[daokoder]

In Dao the easiest would probably be to implement "reactors" using async classes, but they were removed from Dao (though they're still part of documentation). So now the implementation would use channels and parallel functions to achieve similar results.

Actually async classes were not removed. I just updated the concurrent demos, mainly to add explicit "var"s. You can try out "demo/concurrent/async_object.dao".

[dumblob]

In case the game is already available, could you post a link here?

[daokoder]

@dumblob Very sorry for this long delay, I have been very busy with the game development in the past a few months. I finanlly have setup a Steam store page (https://store.steampowered.com/app/1213450/Craftica) for the game a few days ago. I have also setup a simple homepage for this game at http://craftica.net .

The game can compile and run on Linux, but I don't know how to make a distribution-independent release (though I am aware of AppImage, I am not yet familiar with it and haven't got time to try it), so I am afraid it won't be available for testing. I am trying to do my best to release the game before the Chinese New Year day which is less than one month away, so I am on a very short schedule, plus my unfamiliarity with Steam, I am afraid I cannot afford the time to support Linux in its initial release on Steam. I will try to make releases on Linux when I have more time.

[dumblob]

@daokoder thanks for the update. Yeah, just forget about Linux completely until you get some money from buyers on Steam to fund further development. I can test also on Windows, so if you need testing, just make an alpha release on Steam (it's pretty common to do it to gain traction soon enough) and people (including me) can test it.

Btw. AppImage could be the way to go, but it seems the winner of portable software distribution for the Linux world is already now Flatpak (backed by companies like IBM/RedHat, ...).

[daokoder]

@daokoder thanks for the update. Yeah, just forget about Linux completely until you get some money from buyers on Steam to fund further development. I can test also on Windows, so if you need testing, just make an alpha release on Steam (it's pretty common to do it to gain traction soon enough) and people (including me) can test it.

Good to know this, thank you, I will try to make an alpha release on Steam before the first EA release.

Btw. AppImage could be the way to go, but it seems the winner of portable software distribution for the Linux world is already now Flatpak (backed by companies like IBM/RedHat, ...).

Interesting, I will look into this when I have time.

[daokoder]

@dumblob Just let you know that I have just made an alpha release on Steam. Since I didn't manage to make an EA release before the Chinese New Year Day, I decided to postpone it to later this month. With the extra time, I also decided to add Linux support to the alpha and EA release (it turned out a lot easier than I expected).

If you know anyone who is interested in trying this game out, I can provide some Steam keys.

[dumblob]

If you know anyone who is interested in trying this game out, I can provide some Steam keys.

@daokoder I think I actually do - I'm not sure I'll come to this game in the next weeks (too busy 😢), but some of my friends could find some time for testing. I'll contact you per email.

Also, where would you like to get feedback on the game? I've looked quickly at the videos and have few things I noticed 😉 (but this thread might not be the right place though I don't know whether a private email is any better).

[daokoder]

@daokoder I think I actually do - I'm not sure I'll come to this game in the next weeks (too busy 😢), but some of my friends could find some time for testing. I'll contact you per email.

Also, where would you like to get feedback on the game? I've looked quickly at the videos and have few things I noticed 😉 (but this thread might not be the right place though I don't know whether a private email is any better).

Yes, feedbacks are always welcome:). Email would be better for this. Now I mostly use 1650569086 at qq.com.