Creating a framework to be used as a base for many other applications

Oh, interesting, received another retrospective just few days after last one. We have developed a base ASP.NET WebFrom framework named ABC and then developed a series of web applications based on it on 2010/2011. One of them is DEF that is currently (late 2016) in production heavily and I guess will be in production at least till 2021 or even 2025 (no retirement plan yet). DEF is dealing with a database more than 10,000,000 records and is used country wide as a national project. DEF is encountering performance problems and is under constant but small changes of the client.

 

Many companies specially those that are not very technical at managerial level love to write code once but use it in many projects more than once. This is why many IT companies have internal frameworks and many people like myself are searching if it is a good idea at all and if yes, what's the best platform to do that. Creating ABC and making DEF and other applications based on it is a good sample for this approach. I'm gonna to review its real weaknesses as it has been in production for few years now.

 

Large Data

ABC has been used as base project of many applications but no one were dealing with large amount of database records as DEF. At the other hand ABC was not designed to deal with that large amount of data. So DEF has performance issues more than other ABC based projects. Performance issues in turn causing other issues relate to save some records in situations that system load is high.

 

Upgrade

As ABC is base framework and many applications based on it are in production and some of them have multiple instances, so upgrading ABC is hard. Suppose I want to upgrade a component in ABC to enhance some features, this upgrade may cause problems in others, at least I can not be sure that this upgrade is safe for all applications. In DEF case we needed to upgrade Nhibernate and did it in painful and very lengthy process.

slow
slow

Internal mechanism

Like upgrade problem, we have difficulties in changing internal mechanisms and designs. For example changing transaction management is somehow necessary for DEF but it must be done through ABC. And as others are using ABC too, it is not an easy way, and sometimes it is impossible to be accomplished. As a result DEF is forced to live with problems that we know why exists and if DEF was standalone, how can be corrected.

 

Do everything through application channel

For a small application that can be run from a shared host, it is not a bad idea if every operation is done via web application itself. But in a large application like DEF there are situations where other tools are needed. For example we have batch operations that take likely 30 or 60 minutes to complete. A good tool is to use windows services to do this type of work. But DEF uses ASP.NET and IIS to do batches that is not good. Many application pool restarts occur during batch or lengthy operations. Also they reduce speed of current logged users and decrease IIS resources and possibly cause secondary problems. Another example is handling a database table with large record count. We handled it hardly in the application while a better way was to introduce a secondary database and define jobs to move old records to it so main database remaining lighter.

inflexibility
inflexibility

Creating packed mega APS.NET controls

If you are familiar with ASP.NET WebForms you know that we have plenty of ASP.NET controls available there, like drop-down-box. In ABC we have had created some mega controls like them to do bigger operations. Think that they were similar to Telerik or Dundas controls, but larger and wider. For example a grid that was able to do paging, sorting and searching. In theory they were very useful and time saving but they were tightly coupled with other internal structure of the ABC and was very inflexible.

 

Conclusion

General purpose frameworks are seen very promising while not used, but in production many cavities are explored. They are good when used in short term and very very similar applications. If you want speed and flexibility think more about “creating each application from scratch” strategy.

Review structure of a web application that I’m working on

There is a work-flow in Scrum called retrospective. It is about reviewing work done in a sprint. I love the idea, I think talking and communication in a software development team is very important. Inspired from scrum retrospective I'd like to have a review on architecture and design of a project that recently I've been involved in. The project is not finished yet but I think it is a good time to review its structures.

 

The project back-end is implemented by ASP.NET MVC Core and Entity Framework Core and is serving both web API and server backed contents (ASP.NET MVC). Develop is done mostly in Ubuntu but also in Windows too.

copyright https://hakanforss.wordpress.com/2012/04/25/agile-lego-toyota-kata-an-alternative-to-retrospectives/

Projects Structure

While project is not very big and complex and have about 20 tables on the database we decided to have 4 projects. One for domain, dtos and contracts named Domain, another for mainly business logic called Core, another for web project itself that contains cshtmls, controllers and wwwroot directory called Web and another for unit tests called Test. I do agree this is a very common project structure for ASP.NET web applications but I saw not benefit over them except than a simple categorizing that also was achievable with directories. I think it was better to have 2 projects, one for web (combining domain, core and web) and another for test.

 

Interfaces

Programming against interfaces are very popular in C#. It has gained more popularity with wide usage of dependency injection in ASP.NET. Also ASP.NET Core have a built in dependency injection that has increased popularity programming against interfaces. We followed this manner in our project. We create an interface per each service class. We have no mockups in our unit tests so I think using so many interfaces are a bit over-engineering because we have no interface in our project that has more than one implementation. Having large number of interfaces just decreased development velocity. Adding each new method needed changes in two place, service itself and interface that it was implementing.

 

Soft Delete

Soft delete means not deleting database records physically, but instead keep them in database and set a filed in each record named IsDeleted to true. So in the application soft deleted records are not showed or processed. We add this feature to the application so we can track data changes and do not allow any data being really loosed. For this purpose we could have used logging mechanism in a way if a record is deleted a log is added that says who deleted what data and when. Implementing soft delete imposed many manual data integrity check to the application. With delete of each record we must check if any dependent item exists or not, if so prevent deletion.

 

Authorization

I personally made authorization over-engineering. I checked roles in both controllers and services in many cases. My emphasis on separation of controllers and services was a bit high. No entry to the app exists other than MVC controllers and API controllers (they are same in ASP.NET Core). So checking roles in controllers in enough.

 

Using dtos to access data layer

Many application designs allow a direct access to database models. For example controllers action get a MyModel instance directly and pass it to dbSet or services to save it. It is dangerous because ORM dirty checking mechanism may save them in the database in mistake. In this project I used a Dto to pass data to or get data from CRUD services. So controllers are not aware of database models. It increased volume of development but I think it saves us from mysterious data updates in the database.

Update:

Second part of this post can be found here.

Software component re-use in ASP.NET and Django and is it really beneficial?

‌I am an ASP.NET developer for years. In many companies and projects that I worked for there is constant need for re-usable components. Some companies were completely serious about it so based their business on it. When I'm doing software projects on my own management (as project manager or as freelancer) I encounter this subject again.

 

Realistic or not realistic, it is very favorable if we could use previously developed components in new projects. It would be very time saving specially in repetitive projects. Code re-use exists in different levels of ASP.NET. You can use Html helpers or user controls to share components between projects, you also can use services like logger or authentication in several projects. There is a type of component reuse in ASP.NET that is used by modules like ELMAH that is based on HTTP modules or middle wares. None of them are a component re-use that I need. What I need is a complete set of component re-use. I need all core and UI elements all together. For example in logger example, I need core logic and all needed UI all together in a manner that I can plug into a new application so other components of the application can communicate an integrate with it. I know there is a solution for ASP.NET that is called Area that approximately do what I need. It do its re-use in view (UI) well. I just copy files into its directory. But I it no designed a really separate component. It is forced to be aware of mother application's internal functionality specially on database design. Maybe it is the reason that ASP.NET MVC area is not very popular.

 

I've read a lot about Django that is re-use friendly by design. I see it is based on apps. Also I see that there is an app sharing web site for it. But never used it in a real project.

 

By thinking more and more on software re-use (in the context of web development) I realize that not every component re-use is suitable for the application. There is trade-off here. If you want to have a re-usable app then you have to develop it as generic as you can. That itself is causing complexity and creating bug and even consumes more time for development. When you start using a component among several projects you must carefully think of every change you made in the application. Each change must be confirmed as backward compatible as others are using your app. So maintenance would be hard. Apparently this is the reason many web development teams do not employ re-usable components a lot.

 

There is at least one situation that this model of software re-use makes sense. When you produce a re-usable app for a limited range of projects and limited period of time and when you are intended to use your app only in a family of project, that would better suites. Here it is good that Django applications are developed in this manner by default, whether you wan to re-use it or not.

Finding a good front end solution for a semi single page web application

 I am in middle of decision making about what technique or library should I use in front end for a typical web application. Major section of this application is done with ASP.NET MVC with full back end approach. So very little front end development and Ajax calls exists except for cascading drop downs or implementing auto-complets. Every operations are done via server post backs. When you do a CRUD or other operations your request is sent to server, then the result is rendered in server then returned to client and finally is shown. With this manner, front end can not be complicated very much. Pages can not have too many elements and//or too many operations. For large operations, more than one page is needed. A page for main operation then a page for sub-operation that typically is navigated from a main list page.

 

But the problem began from where that some page are wanted to have more than one operation. For example a page for CRUD some models and some sub-pages in it for complementary operations. Supposing that no post-back is allowed here, we would need front end development here. Interacting with DOM and reading/updating them needs plenty jQuery code and also need few server APIs and ajax calls to them. As much as the pages goes larger and need for interaction with user gets higher, for example getting user's confirmation or opening more dialog boxes, volume and complexity of front end code increases. So need of decreasing complexity and development time arises.

 

Here we have 3 options. First, do not allow much front end development and handle all the application with back end MVC only. Front end pages will be simple this way. Pages can not have more than one operation and every operation will cause a post back. Number of total pages will decrease as each single operation need a separate page.

 

Second we can allow multi-operation pages but use no ajax calls. That means jQuery is used to opening dialog boxes and getting user data but instead of using ajax, the form is posted to the server so a post-back occurs. This technique have inflexibility because it not easy to show dialog boxes or getting confirmations from user. Everything is posted to server, then possible errors are detected then error messages are sent back to the client. Also no state can be maintained. After page is returned from server, active controls or even data that was entered in inputs will be lost. Because of these in-flexibilities this technique is not applicable very much.

 

Third technique is to get help from JavaScript libraries and frameworks developed for this problem. This way all functionality we need for front end, including good user interaction, low code complexity and low implement time (except for learning time and setup time) is reached. Cons is learning time, setup time and overhead that it may produce.

 

If we go for third solution, a good choice is to use MVC/MVVM js frameworks that are mostly used for SPAs. Our goal is also is SPA but only for some sections of the web application not all of them. Famous js frameworks for SPAs are Angular.js and Ember.js. But they are too large for our problem. So a smaller one must be selected from several comparisons including this, this and this. Form them I feel that backbone.js (MVP) and knockout.js (MVVM) are better choices. Backbone.js uses more familiar pattern that is MVP and I read somewhere that knockout.js development is slow and community is getting decreasing. So backbone.js could be final choice.

Update

After advocating wiht my friend I decided to add a 4th solution: doing front end manipulations with pure jQuery/ajax code. This code may be lengthy but have less overhead than employing a SPA framework like angular.js or backbone.js.

Update 2

Shawn Wildermuth also did a comparison recently. Find it here.

Selecting a web framework based on reusability and pluggability of components

There are plenty comparison of web frameworks on the Internet. Many of them compare web frameworks in general, some of them compare web frameworks with performance measure. Some are comparing in learning curve, popularity, architecture, speed of development, etc.

 

But I am interested to focus on reusability and pluggability of components. In a web development team it is good to be able to use previously developed portions of project in new projects. For example many projects have membership or accounting section. They can be developed once and used more than once in different projects. You can even think of ticketing or organizational structure management across different separate web projects. Goal is reduce development effort during development mid level web projects.

Reusability

Django is introducing itself as to be so, but how about Rails, ASP.NET, MEAN or other common web frameworks?

 

Django has administrative CRUD interface that can save much time during development. Dhango's moto is “the web framework for perfectionists with deadlines”. Every application in Django is consisted of apps. Each app can implement an independent field of business. Django claims that you can join different apps to to create a complete web application.

 

Django has good documentations but its learning curve is high. It seems efficient for database based applications.

Django is not fully object oriented. It is not as fast as Node.js but does not force you to build everything from scratch. It also seem to have less batteries than Node.js and Rails. Its job market is even smaller than Rails and ASP.NET.

 

Rails is opinionated, so many settings and conventions are set by default. Rails developers can learn it faster and develop with more rapidly. It is not very good at performance but have strong community. Rails is popular in Mac users while. It also is easier while deploying as cloud solutions have better support of it. Its rapid development can compete with reusablity feature that Django claims.

 

Previously I wrote about the subject here, here, here and here.

 

What is your opinions and experiences?