-
Notifications
You must be signed in to change notification settings - Fork 15
Code changes history
-
Solution moved to Visual Studio 2019 and .NET Core 3.1 SDK. Read the SourceCodeOverview.txt for instructions of additonal setup you need to run the tests; you need to create VitaBooksLogs empty database and adjust connection string in WebTests project.
-
Vita.Entities.Emit project is gone - this separate project/assembly is removed, classes moved to the main Vita assembly which directly references the Proxemity package. No more need to set the MyApp.EntityClassProvider property at startup. I realized that this arrangement (separate assembly to handle dependency on Emit) - brings more trouble than it's worth.
-
Logging infrastructure rebuild. Logging to database using Vita.Modules.Logging module and app. You can now direct logs to a dedicated database, see WebTests project for an example of setup. Logging to file - still available, just set the EntityApp.LogPath property. In the plans - logging to other destinations. The sources contain a draft of client for Azure LogAnalytics.
-
Vita.Web package (re-introduced) - integration with ASP.NET-Core 3.1 Web stack. Provides VitaWebMiddleware - a middleware/interceptor for Http pipeline, see WebTests test project for example of setup. Other classes: VitaJwtTokenHandler - provides handling of authentication over HTTP using Jwt tokens; BaseApiController - base case for your API controllers, integrated with the middleware.
Major functionality provided by the middleware:
- Intercepts the HTTP request, sets up an OperationContext and WebContext instances used by API controllers to access the data in the context of the currently logged-in user.
- Handles the ClientFaultException (soft error) from the controller/business logic and translates it into BadRequest returned to the client, with list of faults in the body.
- Logs the request information to the log. Look at WebCallLog table in the VitaBooksLogs database after running the WebTests. For each request you can see complete information, including SQL log - all SQLs that had been fired when processing the request.
Note: Although Vita.Web package and its classes can be used directly in your Web application, I recommend to look at them more like a sample and use its source code as a starting point for your own custom solution; most likely real-world cases would need far more customized solution. But it works AS-IS in most standard scenarios.
-
New Web API packages/projects - API implementations and tests based on Vita.Web. Login module now exposes all its functionality over REST-ful API, implemented in Vita.Modules.Login.Api package. BookStore sample app has API implementation project that is tested (along with Login API) in the WebTests project.
-
WebTests test project. The app starts a REST-ful service for the sample BookStore app, and hits the API endpoints. Shows how to handle login and authenticated users. Uses new separate Arrest package for REST client, available here: https://github.com/rivantsov/arrest
-
Vita.Modules.Legacy package - several entity modules brought back from old 1.9 solution (pre- .NET-core), for backward support of existing apps. The LEGACY token indicates that although the code is fully supported, no future development or enhancement is planned.
Note: currently support is limited to simple sequences - no support for grouping/parenthesis of multiple different operations ( like sql1 UNION sql2 EXCEPT sql3), no precedence/grouping analysis.
It was discovered that when LINQ expression contains several Where calls chained together, the resulting SQL might be incorrect due to lack of proper parenthesis. Fixed, the test TestBugFixes has a snippet to verify the fix.
Patch, no functional changes
- SQLite driver switched to provider/package from SQLite team. Now it is actually double-targeted, you can switch to MS provider as well, see startup code in basic unit test project
- A few fixes and enhancements, fixed several issues in db views
- Include - brought back the facility previously available in v1.9, to run Include queries after the main query to pre-load all related entities for the main query's results. See TestLinqInclude unit test in ExtendedTests project for examples.
- Fixed LINQ Any() method support - this was needed for CanDeleteEntity method that check if an entity has any child records.
- LINQ commands structure refactoring - deep refactoring, main goal - optimization for performance, to avoid any extra work for repeated queries when SQL is already cached. And cleaning up of course.
- Many-to-many lists fix - now runs single JOIN query; this also works for Include queries
- SQL cache (former Query cache) - cache for compiled SQL statements, either LINQ or CRUD; now fully implemented. One of the problems with SQL caching in the current version, compared to old QueryCache, is that CRUD operations now also use dynamically generated SQL. The UPDATE SQL statement updates only columns that were actually changed. It means that SQL cache might contain multiple update SQLs for a table, so the caching key must include the 'mask' representing columns being updated. This required introducing a number of bit masks as property of EntityRecord (data container). And a new BitMask class to support all this. As a result - some refactorings all over the place. The SqlFactory class is a producer of SQL statements - either from SQL cache, reusing identical statement from the past, or building SQL and putting it into cache.
- Oracle Schemas - adding support to Oracle driver for 'schemas' (actually USERs in Oracle terms); it appears it is standard practice in Oracle databases to create 'users' that act as schemas and contain collections of tables, views etc.
- Miscellaneous fixes, improvements, refactorings
Next - bringing back support for LINQ Include functionality.
Oracle is now available in VITA.
To sum it up - Oracle is quite a trouble to program against... Ending semicolon in standalone SQLs - part of standard for decades, but still causes error in Oracle... INFORMATION_SCHEMA views are not available. INSERT syntax for multiple rows is not available. 'Empty' database comes with hundreds of system tables and views, and it is really impossible to reliably separate these from your app tables. This has quite an impact on DB upgrade facilities, as you can imagine. There are no ints, only decimals, and no way to safely read the decimal - Oracle has larger decimal precision than .NET decimal, so reading DbReader value might blow up, and no remedy - people complaining for years...
And many many other things...
Oracle type system is also something special. But one positive outcome is that VITA drivers type system (DB types handling) was completely redesigned - and finally cleaned up and is much much better now, I believe.
So Oracle is here, enjoy it. Now VITA supports all major database servers, so it's time to start writing modules and applications that do real stuff.
The Framework moved to .NET Standard 2.0. Not all components and features are ported, see below for a list of these. The biggest change is abandoning use of stored procedures as a primary way to do CRUD operations. Everything is dynamic SQL now, batched in one multi-statement SQL in case of multiple updates. This switch required substantial rewrite of SQL-generating facilities - mainly to avoid multiple string concatenations and producing memory garbage. It is in fact a SQL templating engine.
In the earlier version using batch mode was possible only with stored procedures; now it is possible with plain SQL - so you can use efficient batch ops even if you do not full control over the database and cannot create procs there.
Among extra benefits:
- INSERTs: Using special form of Insert statement with multiple records to insert. VITA automatically groups records by table/operation, and then merges multiple records to insert for one table into single INSERT.
- DELETEs: Deleting multiple records with single DELETE statement, using from 'WHERE Id IN (@id-list)' where Id-list is passed as parameter or as literal list. VITA also does this grouping automatically.
- UPDATEs: Updating only the columns that were actually changed, handled automatically.
Features/Components to be ported or re-implemented from version 1.9:
- Authorization Framework
- Query cache
- Data cache - to be re-implemented
- Query-with-Includes - pre-loading related records in queries, known as "N+1" queries problem
- Logging modules and infrastructure
- Other useful modules
- Web-API stack integration
- Refactored background execution of jobs in JobExecutionModule - aligned with ASP.NET. Running long background tasks is tricky under ASP.NET: the engine makes a cleanup when Web request completes - it kills/aborts all spawned threads. So it should be run through a special method in HostingEnvironment class. Additional challenge is running unit tests (with self-hosting option) - the HostingEnvironment's method does not work, but cleanup behavior is still there. For this case I use a simple ugly hack - pass it to a timer event. Design: there is a new service IBackgroundTaskService - to run background task. By default it is trivial (Task.Run(...)); for Web environment it is replaced with a little more advanced implementation that uses either HostingEnvironment.QueueBackgroundWorkItem, or a simple timer redirection (for unit test self-host). This area and JobExecutionModule need more attention and refactoring - will do when moved to .NET Core.
Minor bugs update, with some refactoring in preparation for .NET Core version.
- Encryption services and EncryptedDataModule - removing dependencies on it from other modules, and recommend you doing the same. Admittedly not the best design - a single table with encrypted values, with clear chance of orhan records appearing there over time. Changes: LoginModule, OAuthClientModule no longer use EncryptedData module, values (emails for password recovery, OAuth tokens) are stored unencrypted directly in corresponding tables - I think it's OK, not worth the complexity of referencing a separate record in EncryptedData table. There is a migration script provided (automatic), so if you have old data, it will unencrypt it and copy values to new fields. If you use EncryptedData module somewhere else, I recommend doing the same. In the future the module with EncryptedData table will still be available, but encryption service will be a general service, independent of this module. One of the reasons I am doing this is that in .NET Core the situation with available algorithms is not quite clear yet (at some point RinjdaelManaged - default for Fx - was not in .NET Core). So I do not want you to end up with encrypted data in the database with no way to unencrypt it when we move to .NET Core - so let's unencrypt it before it's too late.
- Bug fix - handling SmallDateTime columns in LINQ for MS SQL Server
- Hashing method for hashing DB Views sources changed from plain MD5 to SHA256 - to comply with FIPS; does not change anything externally.
- Fixed nuget dependencies (versions) in drivers packages
- Some other minor fixes.
I am continuing with .NET Core migration. The reason it takes so long - slowly drowned into a big refactoring, for BIIIG improoooovements - well, you know, we all do this. I promise, it will be out soon.
- The biggest new thing is the Job Execution Module. More information here: https://github.com/rivantsov/vita/blob/master/3.Modules/Vita.Modules/JobExecution/AboutJobEngine.txt
- Upgraded data providers packages to newest available versions. Except Postgres provider, could not make it work with the latest, upgraded to highest 'working' version.
- LoggingApp - allow enable modules/logs selectively.
-
Login module - multiple minor changes; if you use this module and/or its API, make sure you double-check all places that might be impacted:
UserName is not Nullable anymore in Login module. Good news - all methods in Login API are now commented, you can see all information in Swagger (in BooksUI app). LoginInfo object used in Login API is refactored - old bool fields are gone, now only bit flags. URL structure in Rest API for multi-factor login is refactored - look at Swagger page. Refresh token for user session is now returned in LoginResponse - you can use it to refresh session token before it expires. PasswordResetController.StartProcess now returns 'boxed' object with string value inside (see comments on the method for explanation). Bug fix - second login without logout was returning old session token, not new one. Pins and temp password now contain only 'safe' chars - excluding those that look alike (ex: 0 and letter O). New API endpoint to submit point for email verification - not requiring user to be logged in; use it in direct-hit link in email. And some other minor changes in Login module/API. - EntityApp.ActivationLog - refactored and renamed into SystemLog. EntityApp.ActivationLogPath is deprecated, use SystemLogPath property - assign file name (or full path) to persist system log. System log contains system messages about system startup/shutdown. ErrorLog fixed - in case of failure to write to database, writes to SystemLog, not Windows EventLog (causes permission problems). So in Web apps, make sure app can write to file located at SystemLogPath value.
- SQlite driver refactored, now relies mostly on default formatting/conversions by SQLite.NET provider. Should fix reported problems of reading existing databases.
- New option DbOptions.AddSchemaToTableName - for servers like SQlite that do not support schemas; for SQlite the flag is set by default. With this flag, schema is prefixed to table names: mysch_MyTable. Allows better management of big models.
- OAuthClientModule refactoring/fixes. Added Authorized event fired when OAuth process is completed successfully. Fixed OAuthDemoApp.
- WebApiClient error handling refactoring; ErrorHandler class/property is removed; new event Error added, to allow custom error processing.
- UpdateQuery enhancement - now correctly supports skip, take, order-by clauses.
- DisplayAttribute fixed (and much improved the code), now handles more cases correctly
- New attribute PropagateUpdatedOn - put it on reference property (IBookOrder.Order), and each change of 'this' child entity will cause UpdatedOn property on parent entity (BookOrder) to update, as if the parent was modified itself. Allows tracking of 'last-changed-date' on doc header, including changes to child records ONLY.
- Added EntitySession.TransactionTags property - a dictionary that can store extra keys/values, cleared automatically on transaction end. Can be used for things like tracking added extra operations in automatic handlers, to avoid doing it multiple times. Used in PropagateUdatedOn attribute.
- Setting string column to single space - fixed, now non-nullable string column accepts single space value. (but not string.Empty)
- Allowing foreign key columns to be declared explicitly - now does not fail if a column like 'Order_Id' is declared explicitly next to IOrderLine.Order property. Previously was failing because .Order property was trying to create underlying column Order_Id.
- Bug fix - calculated property with both Get and Set was causing error (hanging) at startup; now fixed.
- SearchParams.DefaultIfNull helper method (used in construction of search queries) now has maxRows parameters; allows you to limit max number of rows to return.
- Preparing for .NET Core - multiple refactorings to prepare for .NET Core migration - next release for sure. ObjectCache is rewritten, no longer uses MemoryCache object.
Next release main focus - .NET Core.
Many bug fixes, refactorings; WebClientLog - new log module
- WebClientLogModule - for logging all Web client calls (HttpClient). Essentially logs all calls to external APIs. WebApiClient class automatically logs all call into this log/table. The module is part of LoggingApp.
- Bug fix: Views requiring Order-By clause - fixed.
- Async-sync bridge - refactored - the AsyncHelper class refactored; now much simpler implementation of sync/async bridge.
- Support for OrderBy attribute on list properties - now the attribute actually works in these places. PersistOrderIn attribute also sets OrderBy automatically.
- Fixed resetting database scenario in unit tests - discovered that Extended tests do not work with config option 'ResetDatabase=true'. It even can corrupt installation of the server (MySql). The problem was that the operation was trying to Drop too much - including system tables. Now fixed. Advice - do not run unit test using DB server's root account.
- Extra login options - the login info object sent to Login API endpoint now has a property ExpirationType - the client can decide what kind of expiration to use. There is also a new option - LongFixedTerm; a new method/API endpoint - RefreshToken, to get new token before old one expires
- Refactoring for .NET Core - removed dependency on System.Data.DataSet, .NET Core version does not have it.
Patch release, mostly bug fixes and minor improvements
- MS SQL: XX_ArrayById procs fix - added CAST expression to generated stored procs, those selecting by array of IDs. Previously was used in LINQ, but forgot to add this to auto-generated procedures
- MS SQL: added READ_COMMITTED_SNAPSHOT=ON script to enable snapshot isolation; turns out this must be done too, to handle locks properly
- WebApiClient - added mapping options - now you can map PascalCase properties in models to all-lower-with-underscore names in JSON. Ex: FirstName <-> first_name; mapping happens automatically (used this in MailChimp client, makes life a bit easier)
This update was driven primarily by current needs of the app that we build at my workplace, so it happens to be more about Web API features rather than ORM/database things.
- Swagger - support for Swagger API document pages. Run sample BookStore UI app and click on Api Docs link on the left - you will see Swagger UI with 'documentation' for all available controller endpoints. Xml comments are integrated also, look at Login controller, one of the methods shows a doc line. Something to be completed. Look at SwaggerConfig.cs file for the ways to set it up. One change compared to default configuration by Swashbuckle: the setup method is called explicitly (not thru web-init attribute at assembly level) - in this case it must be called before Route config method. Endpoints are grouped by controller name, but you can set group explicitly with ApiGroup attribute.
- OAuth 2.0 Client (re-)implementation. OAuthClientModule was rebuilt to support full OAuth 2.0 flow. A new demo/test app in samples folder: OAuthClientDemoApp. The module pre-installs the info about some popular OAuth servers (Google, WindowsLive, Facebook, etc), including their 'specifics' expressed as flags. Run the demo app and see full OAuth cycle for each server. Note - you have to provide your own app registration info with target servers; see instructions there right in the app. The module comes with set of API controllers that can be used directly in a web app.
- DataHistory module - a simple, almost primitive way to save history of changes for a table. BookStore app now uses this module to track every change (and all past versions) of BookReview entities. Look at unit test for the feature to see how it works. Properties for previous versions are saved in a dictionary serialized in a text column.
- DataAccess service refactored a bit - IDataSourceManagementService is no longer there, all methods are in IDataAccessService, which is available directly in EntityApp.DataAccess. For convenience of dynamically hooking up databases (data sources) on the run, there is an event now DataSourceAdd which is fired when data source specified by name in OperationContext is missing - an app code in event handler can dynamically hookup a new data source. Those who use multi-tenant scenario would need to adjust their code a bit.
- OneToOne attribute - to construct back-ref properties for one-to-one scenarios (when foreign key in child entity is also a primary key in child). Example - look at party module. IOrg entity references IParty (and it is PK also); so IParty has Org property with OneToOne attribute - this is a reference without underlying column, derived from back ref IOrg.Party -> Party.
- New options to disable stored procs and batch mode in entity session - used in some modules like DbInfo and ErrorLog, that might need to execute db commands when stored procs are 'not ready yet'.
- LINQ queires with array parameters - refinement for MS SQL, now queries have explicit cast to target type in sub-select. The goal is to avoid confusing query optimizer that was picking up index scan instead of seek without this CAST.
-
Multiple subtle bug fixes:
- Bug in computed property, event firing for DependsOn properties;
- Fixed DateTime parameter 'formatting' in LINQ queries for SQLite;
- Writing wrong current version in DbUpgrageLog module (was writing LoggingApp version, not the main app version)
- Detecting CascadeDelete change attribute on a property - previously was not catching it when it was the only change.