-
Notifications
You must be signed in to change notification settings - Fork 15
Code changes history
- Custom LINQ functions implemented as SQL expressions .
- DB-computed members (generated columns), based on SQL expressions)
You can now define a static function (extension method) that you can use in LINQ queries against the database, and it will be implemented as a custom SQL expression inside the final SELECT query. You define SQL expressions, one per each server type, using SqlExpression attribute. Here is a DbAddYears function: https://github.com/rivantsov/vita/blob/476983f2b5927da8207c4d5fd06e17ebce72091b/src/6.UnitTests/Vita.Testing.BasicTests/MiscTests_Model.cs#L138
and here how you use it: https://github.com/rivantsov/vita/blob/476983f2b5927da8207c4d5fd06e17ebce72091b/src/6.UnitTests/Vita.Testing.BasicTests/MiscTests.cs#L363
DB-computed members are similar - an entity member that has an underlying SQL Expression. There are 3 types of DB-computed members:
- Expression only - there is no DB column, entity member value is computed from a give SQL snippet in SELECTs
- Column (aka Virtual Column) - column in db table without actual storage, it is calculated by database on the fly
- Stored column - the column is computed on create/update and the value is stored in the table row.
Not all servers support both virtual and stored columns. Oracle does not support Stored columns; Postgres does not support Virtual. Additional trouble is with DB upgrade. Servers parse and reformat the expression when creating generated columns, so detecting diff db/entities is a challenge. For MS SQL, MySql, Postgres - start with custom expr, start the app, look at reformatted expr, and replace the value in the attribute with the value now in the database. For your convenience, VITA prints out both expressions to Debug output when DB upgrade detects the difference.
For Oracle, things are much worse. Expression is available but it is stored in LONG column - Oracle deprecated LONG data type long ago, and currently there is no way to read the value with .NET Oracle provider. So for Oracle, it's all or nothing - there's a static flag in OracleDbDriver that controls this.
Here is an example of DB computed columns:
The biggest addition is a first draft of GraphQL integration - and a sample GraphQL server with ASP.NET Core and NGraphQL. The majority of the GraphQL service is outside VITA code. But there is one advanced feature added in VITA to support GraphQL: the SmartLoad facility. It is a solution for the (N+1) problem in GraphQL, but I see it absolutely useful in general, outside of GraphQL context. SmartLoad is about automatically batch-loading the related entities, just like with Include's, but without explicitly specifying the Include expressions in the original query. I will write a separate doc about this feature and how it works.
Other fixes:
- Fix for problem of loading list properties with includes, when empty lists results were not registered as empty, and caused repeated db calls to load them again.
- SelectMany LINQ method worked incorrectly, returning nulls instead of objects; now fixed.
- For many-to-many list properties, the link entity is now available through EntityHelper.GetLinkEntities help method. This makes it possible to have extra properties on link entities.
- For Web API: Jwt token info is now available in OperationContext under keys defined in VitaJwtTokenHandler class.
- For MS SQL Server the provider is changed to Microsoft.Data.SqlClient (from System.Data.SqlClient). See https://www.sqlservercentral.com/blogs/moving-to-a-new-sqlclient for more info. Important: if you reference the SqlClient classes directly in your code, you need to switch to this package as well.
- Restored NoColumn attribute functionality - the attribute marks entity property that has no underlying column, it is in-memory only value in entity.
- Internal changes to projects - removed AssemblyInfo.cs files, switched to using only info in .csproj files; latest recommended arrangement.
- Fix - Logging to file in Web app
- Fix: Aggregate over int column throwing exception
- Fix for issue 146 - exception thrown on entity ref assignment
- Hooked AppEvents.Connected event
- Fixed use of default size in Size attr (issue #132)
- Added ILogFileWriter interface (issue #121)
- Fixed CreatePrimaryKey method for composite-over-composite keys
- Fixed LINQ list.Contains(long values) expressions in MS SQL Server for long (BIGINT) columns/lists.
-
Solution moved to Visual Studio 2019 and .NET Core 3.1 SDK. Read the SourceCodeOverview.txt for instructions of additonal setup you need to run the tests; you need to create VitaBooksLogs empty database and adjust connection string in WebTests project.
-
Vita.Entities.Emit project is gone - this separate project/assembly is removed, classes moved to the main Vita assembly which directly references the Proxemity package. No more need to set the MyApp.EntityClassProvider property at startup. I realized that this arrangement (separate assembly to handle dependency on Emit) - brings more trouble than it's worth.
-
Logging infrastructure rebuild. Logging to database using Vita.Modules.Logging module and app. You can now direct logs to a dedicated database, see WebTests project for an example of setup. Logging to file - still available, just set the EntityApp.LogPath property. In the plans - logging to other destinations. The sources contain a draft of client for Azure LogAnalytics.
-
Vita.Web package (re-introduced) - integration with ASP.NET-Core 3.1 Web stack. Provides VitaWebMiddleware - a middleware/interceptor for Http pipeline, see WebTests test project for example of setup. Other classes: VitaJwtTokenHandler - provides handling of authentication over HTTP using Jwt tokens; BaseApiController - base case for your API controllers, integrated with the middleware.
Major functionality provided by the middleware:
- Intercepts the HTTP request, sets up an OperationContext and WebContext instances used by API controllers to access the data in the context of the currently logged-in user.
- Handles the ClientFaultException (soft error) from the controller/business logic and translates it into BadRequest returned to the client, with list of faults in the body.
- Logs the request information to the log. Look at WebCallLog table in the VitaBooksLogs database after running the WebTests. For each request you can see complete information, including SQL log - all SQLs that had been fired when processing the request.
Note: Although Vita.Web package and its classes can be used directly in your Web application, I recommend to look at them more like a sample and use its source code as a starting point for your own custom solution; most likely real-world cases would need far more customized solution. But it works AS-IS in most standard scenarios.
-
New Web API packages/projects - API implementations and tests based on Vita.Web. Login module now exposes all its functionality over REST-ful API, implemented in Vita.Modules.Login.Api package. BookStore sample app has API implementation project that is tested (along with Login API) in the WebTests project.
-
WebTests test project. The app starts a REST-ful service for the sample BookStore app, and hits the API endpoints. Shows how to handle login and authenticated users. Uses new separate Arrest package for REST client, available here: https://github.com/rivantsov/arrest
-
Vita.Modules.Legacy package - several entity modules brought back from old 1.9 solution (pre- .NET-core), for backward support of existing apps. The LEGACY token indicates that although the code is fully supported, no future development or enhancement is planned.
Note: currently support is limited to simple sequences - no support for grouping/parenthesis of multiple different operations ( like sql1 UNION sql2 EXCEPT sql3), no precedence/grouping analysis.
It was discovered that when LINQ expression contains several Where calls chained together, the resulting SQL might be incorrect due to lack of proper parenthesis. Fixed, the test TestBugFixes has a snippet to verify the fix.
Patch, no functional changes
- SQLite driver switched to provider/package from SQLite team. Now it is actually double-targeted, you can switch to MS provider as well, see startup code in basic unit test project
- A few fixes and enhancements, fixed several issues in db views
- Include - brought back the facility previously available in v1.9, to run Include queries after the main query to pre-load all related entities for the main query's results. See TestLinqInclude unit test in ExtendedTests project for examples.
- Fixed LINQ Any() method support - this was needed for CanDeleteEntity method that check if an entity has any child records.
- LINQ commands structure refactoring - deep refactoring, main goal - optimization for performance, to avoid any extra work for repeated queries when SQL is already cached. And cleaning up of course.
- Many-to-many lists fix - now runs single JOIN query; this also works for Include queries
- SQL cache (former Query cache) - cache for compiled SQL statements, either LINQ or CRUD; now fully implemented. One of the problems with SQL caching in the current version, compared to old QueryCache, is that CRUD operations now also use dynamically generated SQL. The UPDATE SQL statement updates only columns that were actually changed. It means that SQL cache might contain multiple update SQLs for a table, so the caching key must include the 'mask' representing columns being updated. This required introducing a number of bit masks as property of EntityRecord (data container). And a new BitMask class to support all this. As a result - some refactorings all over the place. The SqlFactory class is a producer of SQL statements - either from SQL cache, reusing identical statement from the past, or building SQL and putting it into cache.
- Oracle Schemas - adding support to Oracle driver for 'schemas' (actually USERs in Oracle terms); it appears it is standard practice in Oracle databases to create 'users' that act as schemas and contain collections of tables, views etc.
- Miscellaneous fixes, improvements, refactorings
Next - bringing back support for LINQ Include functionality.
Oracle is now available in VITA.
To sum it up - Oracle is quite a trouble to program against... Ending semicolon in standalone SQLs - part of standard for decades, but still causes error in Oracle... INFORMATION_SCHEMA views are not available. INSERT syntax for multiple rows is not available. 'Empty' database comes with hundreds of system tables and views, and it is really impossible to reliably separate these from your app tables. This has quite an impact on DB upgrade facilities, as you can imagine. There are no ints, only decimals, and no way to safely read the decimal - Oracle has larger decimal precision than .NET decimal, so reading DbReader value might blow up, and no remedy - people complaining for years...
And many many other things...
Oracle type system is also something special. But one positive outcome is that VITA drivers type system (DB types handling) was completely redesigned - and finally cleaned up and is much much better now, I believe.
So Oracle is here, enjoy it. Now VITA supports all major database servers, so it's time to start writing modules and applications that do real stuff.
The Framework moved to .NET Standard 2.0. Not all components and features are ported, see below for a list of these. The biggest change is abandoning use of stored procedures as a primary way to do CRUD operations. Everything is dynamic SQL now, batched in one multi-statement SQL in case of multiple updates. This switch required substantial rewrite of SQL-generating facilities - mainly to avoid multiple string concatenations and producing memory garbage. It is in fact a SQL templating engine.
In the earlier version using batch mode was possible only with stored procedures; now it is possible with plain SQL - so you can use efficient batch ops even if you do not full control over the database and cannot create procs there.
Among extra benefits:
- INSERTs: Using special form of Insert statement with multiple records to insert. VITA automatically groups records by table/operation, and then merges multiple records to insert for one table into single INSERT.
- DELETEs: Deleting multiple records with single DELETE statement, using from 'WHERE Id IN (@id-list)' where Id-list is passed as parameter or as literal list. VITA also does this grouping automatically.
- UPDATEs: Updating only the columns that were actually changed, handled automatically.
Features/Components to be ported or re-implemented from version 1.9:
- Authorization Framework
- Query cache
- Data cache - to be re-implemented
- Query-with-Includes - pre-loading related records in queries, known as "N+1" queries problem
- Logging modules and infrastructure
- Other useful modules
- Web-API stack integration
- Refactored background execution of jobs in JobExecutionModule - aligned with ASP.NET. Running long background tasks is tricky under ASP.NET: the engine makes a cleanup when Web request completes - it kills/aborts all spawned threads. So it should be run through a special method in HostingEnvironment class. Additional challenge is running unit tests (with self-hosting option) - the HostingEnvironment's method does not work, but cleanup behavior is still there. For this case I use a simple ugly hack - pass it to a timer event. Design: there is a new service IBackgroundTaskService - to run background task. By default it is trivial (Task.Run(...)); for Web environment it is replaced with a little more advanced implementation that uses either HostingEnvironment.QueueBackgroundWorkItem, or a simple timer redirection (for unit test self-host). This area and JobExecutionModule need more attention and refactoring - will do when moved to .NET Core.
Minor bugs update, with some refactoring in preparation for .NET Core version.
- Encryption services and EncryptedDataModule - removing dependencies on it from other modules, and recommend you doing the same. Admittedly not the best design - a single table with encrypted values, with clear chance of orhan records appearing there over time. Changes: LoginModule, OAuthClientModule no longer use EncryptedData module, values (emails for password recovery, OAuth tokens) are stored unencrypted directly in corresponding tables - I think it's OK, not worth the complexity of referencing a separate record in EncryptedData table. There is a migration script provided (automatic), so if you have old data, it will unencrypt it and copy values to new fields. If you use EncryptedData module somewhere else, I recommend doing the same. In the future the module with EncryptedData table will still be available, but encryption service will be a general service, independent of this module. One of the reasons I am doing this is that in .NET Core the situation with available algorithms is not quite clear yet (at some point RinjdaelManaged - default for Fx - was not in .NET Core). So I do not want you to end up with encrypted data in the database with no way to unencrypt it when we move to .NET Core - so let's unencrypt it before it's too late.
- Bug fix - handling SmallDateTime columns in LINQ for MS SQL Server
- Hashing method for hashing DB Views sources changed from plain MD5 to SHA256 - to comply with FIPS; does not change anything externally.
- Fixed nuget dependencies (versions) in drivers packages
- Some other minor fixes.
I am continuing with .NET Core migration. The reason it takes so long - slowly drowned into a big refactoring, for BIIIG improoooovements - well, you know, we all do this. I promise, it will be out soon.
- The biggest new thing is the Job Execution Module. More information here: https://github.com/rivantsov/vita/blob/master/3.Modules/Vita.Modules/JobExecution/AboutJobEngine.txt
- Upgraded data providers packages to newest available versions. Except Postgres provider, could not make it work with the latest, upgraded to highest 'working' version.
- LoggingApp - allow enable modules/logs selectively.
-
Login module - multiple minor changes; if you use this module and/or its API, make sure you double-check all places that might be impacted:
UserName is not Nullable anymore in Login module. Good news - all methods in Login API are now commented, you can see all information in Swagger (in BooksUI app). LoginInfo object used in Login API is refactored - old bool fields are gone, now only bit flags. URL structure in Rest API for multi-factor login is refactored - look at Swagger page. Refresh token for user session is now returned in LoginResponse - you can use it to refresh session token before it expires. PasswordResetController.StartProcess now returns 'boxed' object with string value inside (see comments on the method for explanation). Bug fix - second login without logout was returning old session token, not new one. Pins and temp password now contain only 'safe' chars - excluding those that look alike (ex: 0 and letter O). New API endpoint to submit point for email verification - not requiring user to be logged in; use it in direct-hit link in email. And some other minor changes in Login module/API. - EntityApp.ActivationLog - refactored and renamed into SystemLog. EntityApp.ActivationLogPath is deprecated, use SystemLogPath property - assign file name (or full path) to persist system log. System log contains system messages about system startup/shutdown. ErrorLog fixed - in case of failure to write to database, writes to SystemLog, not Windows EventLog (causes permission problems). So in Web apps, make sure app can write to file located at SystemLogPath value.
- SQlite driver refactored, now relies mostly on default formatting/conversions by SQLite.NET provider. Should fix reported problems of reading existing databases.
- New option DbOptions.AddSchemaToTableName - for servers like SQlite that do not support schemas; for SQlite the flag is set by default. With this flag, schema is prefixed to table names: mysch_MyTable. Allows better management of big models.
- OAuthClientModule refactoring/fixes. Added Authorized event fired when OAuth process is completed successfully. Fixed OAuthDemoApp.
- WebApiClient error handling refactoring; ErrorHandler class/property is removed; new event Error added, to allow custom error processing.
- UpdateQuery enhancement - now correctly supports skip, take, order-by clauses.
- DisplayAttribute fixed (and much improved the code), now handles more cases correctly
- New attribute PropagateUpdatedOn - put it on reference property (IBookOrder.Order), and each change of 'this' child entity will cause UpdatedOn property on parent entity (BookOrder) to update, as if the parent was modified itself. Allows tracking of 'last-changed-date' on doc header, including changes to child records ONLY.
- Added EntitySession.TransactionTags property - a dictionary that can store extra keys/values, cleared automatically on transaction end. Can be used for things like tracking added extra operations in automatic handlers, to avoid doing it multiple times. Used in PropagateUdatedOn attribute.
- Setting string column to single space - fixed, now non-nullable string column accepts single space value. (but not string.Empty)
- Allowing foreign key columns to be declared explicitly - now does not fail if a column like 'Order_Id' is declared explicitly next to IOrderLine.Order property. Previously was failing because .Order property was trying to create underlying column Order_Id.
- Bug fix - calculated property with both Get and Set was causing error (hanging) at startup; now fixed.
- SearchParams.DefaultIfNull helper method (used in construction of search queries) now has maxRows parameters; allows you to limit max number of rows to return.
- Preparing for .NET Core - multiple refactorings to prepare for .NET Core migration - next release for sure. ObjectCache is rewritten, no longer uses MemoryCache object.
Next release main focus - .NET Core.
Many bug fixes, refactorings; WebClientLog - new log module
- WebClientLogModule - for logging all Web client calls (HttpClient). Essentially logs all calls to external APIs. WebApiClient class automatically logs all call into this log/table. The module is part of LoggingApp.
- Bug fix: Views requiring Order-By clause - fixed.
- Async-sync bridge - refactored - the AsyncHelper class refactored; now much simpler implementation of sync/async bridge.
- Support for OrderBy attribute on list properties - now the attribute actually works in these places. PersistOrderIn attribute also sets OrderBy automatically.
- Fixed resetting database scenario in unit tests - discovered that Extended tests do not work with config option 'ResetDatabase=true'. It even can corrupt installation of the server (MySql). The problem was that the operation was trying to Drop too much - including system tables. Now fixed. Advice - do not run unit test using DB server's root account.
- Extra login options - the login info object sent to Login API endpoint now has a property ExpirationType - the client can decide what kind of expiration to use. There is also a new option - LongFixedTerm; a new method/API endpoint - RefreshToken, to get new token before old one expires
- Refactoring for .NET Core - removed dependency on System.Data.DataSet, .NET Core version does not have it.
Patch release, mostly bug fixes and minor improvements
- MS SQL: XX_ArrayById procs fix - added CAST expression to generated stored procs, those selecting by array of IDs. Previously was used in LINQ, but forgot to add this to auto-generated procedures
- MS SQL: added READ_COMMITTED_SNAPSHOT=ON script to enable snapshot isolation; turns out this must be done too, to handle locks properly
- WebApiClient - added mapping options - now you can map PascalCase properties in models to all-lower-with-underscore names in JSON. Ex: FirstName <-> first_name; mapping happens automatically (used this in MailChimp client, makes life a bit easier)
This update was driven primarily by current needs of the app that we build at my workplace, so it happens to be more about Web API features rather than ORM/database things.
- Swagger - support for Swagger API document pages. Run sample BookStore UI app and click on Api Docs link on the left - you will see Swagger UI with 'documentation' for all available controller endpoints. Xml comments are integrated also, look at Login controller, one of the methods shows a doc line. Something to be completed. Look at SwaggerConfig.cs file for the ways to set it up. One change compared to default configuration by Swashbuckle: the setup method is called explicitly (not thru web-init attribute at assembly level) - in this case it must be called before Route config method. Endpoints are grouped by controller name, but you can set group explicitly with ApiGroup attribute.
- OAuth 2.0 Client (re-)implementation. OAuthClientModule was rebuilt to support full OAuth 2.0 flow. A new demo/test app in samples folder: OAuthClientDemoApp. The module pre-installs the info about some popular OAuth servers (Google, WindowsLive, Facebook, etc), including their 'specifics' expressed as flags. Run the demo app and see full OAuth cycle for each server. Note - you have to provide your own app registration info with target servers; see instructions there right in the app. The module comes with set of API controllers that can be used directly in a web app.
- DataHistory module - a simple, almost primitive way to save history of changes for a table. BookStore app now uses this module to track every change (and all past versions) of BookReview entities. Look at unit test for the feature to see how it works. Properties for previous versions are saved in a dictionary serialized in a text column.
- DataAccess service refactored a bit - IDataSourceManagementService is no longer there, all methods are in IDataAccessService, which is available directly in EntityApp.DataAccess. For convenience of dynamically hooking up databases (data sources) on the run, there is an event now DataSourceAdd which is fired when data source specified by name in OperationContext is missing - an app code in event handler can dynamically hookup a new data source. Those who use multi-tenant scenario would need to adjust their code a bit.
- OneToOne attribute - to construct back-ref properties for one-to-one scenarios (when foreign key in child entity is also a primary key in child). Example - look at party module. IOrg entity references IParty (and it is PK also); so IParty has Org property with OneToOne attribute - this is a reference without underlying column, derived from back ref IOrg.Party -> Party.
- New options to disable stored procs and batch mode in entity session - used in some modules like DbInfo and ErrorLog, that might need to execute db commands when stored procs are 'not ready yet'.
- LINQ queires with array parameters - refinement for MS SQL, now queries have explicit cast to target type in sub-select. The goal is to avoid confusing query optimizer that was picking up index scan instead of seek without this CAST.
-
Multiple subtle bug fixes:
- Bug in computed property, event firing for DependsOn properties;
- Fixed DateTime parameter 'formatting' in LINQ queries for SQLite;
- Writing wrong current version in DbUpgrageLog module (was writing LoggingApp version, not the main app version)
- Detecting CascadeDelete change attribute on a property - previously was not catching it when it was the only change.