I had this error with my custom behavior extension that I wanted to add as an endpoint behavior. So, I edited the schema used in Visual Studio 2017 to get rid of the warning in my web.config file. It's the same warning that you received:
The element 'behavior' has invalid child element 'CustomSecurity'. List of possible elements expected: 'clientVia, callbackDebug, callbackTimeouts, clear, clientCredentials, transactedBatching, dataContractSerializer, dispatcherSynchronization, remove, synchronousReceive, webHttp, enableWebScript, endpointDiscovery, soapProcessing'.
My web.config has:
<system.serviceModel>
<extensions>
<behaviorExtensions>
<add name="CustomSecurity"
type="FullyQualifiedPath.MyCustomBehaviorExtension, MyAssemblyName"/>
</behaviorExtensions>
</extensions>
<endpointBehaviors>
<behavior name="CustomServiceBehavior">
<CustomSecurity />
</behavior>
</endpointBehaviors>
<endpoint address="https://SomeServer/MyService.svc/soap"
behaviorConfiguration="CustomServiceBehavior" binding="basicHttpBinding"
bindingConfiguration="BasicHttpBinding_IProject" contract="ProjectService.IProject"
name="BasicHttpBinding_IProject" />
The CustomSecurity XML node always had the blue squiggly line underneath it in Visual Studio. It shows up as a Warning in the Error List window. I wanted to get rid of it because each time I tried to update a Service Reference, it would fail because of the warning in web.config.
So, to fix it, you'll need to edit the Schema that Visual Studio uses to validate elements. So, I opened my web.config, then selected XML on the main Visual Studio menu bar. Then select Schemas. You'll get a long list of schemas. Find "DotNetConfig.xsd" (or DotNetConfig[XX].xsd where XX is the .NET Framework version) as seen below.
Update: you might want to edit any/all xsd files with the DotNetConfig file prefix. Typically you do not want to use all of the DotNetConfigXX.xsd files at once. It's best to have the "Use this schema" option (in the Use column) turned on for only one; otherwise, you might see errors about schema elements already being defined.
Browse to the path shown in the Location column and edit the xsd file. Search for:<xs:element name="behavior" vs:help="configuration/system.serviceModel/behaviors/endpointBehaviors/behavior">
Then add a new xs:element node within the xs:choice node with the name of your custom behavior extension; in my case, CustomSecurity. Save the file and Visual Studio should validate against the new schema automatically and you should not get a warning in your web.config any longer.
<xs:element name="behavior" vs:help="configuration/system.serviceModel/behaviors/endpointBehaviors/behavior">
<xs:complexType>
<xs:annotation>
<xs:documentation>The behavior element contains a collection of settings for the behavior of an endpoint.</xs:documentation>
</xs:annotation>
<xs:choice minOccurs="0" maxOccurs="unbounded">
<xs:element name="CustomSecurity" vs:help="configuration/system.serviceModel/behaviors/endpointBehaviors/behavior/CustomSecurity">
<xs:complexType>
<xs:annotation>
<xs:documentation>Specifies the behavior extension class applied to the endpoint.</xs:documentation>
</xs:annotation>
<xs:anyAttribute namespace="http://schemas.microsoft.com/XML-Document-Transform" processContents="strict" />
</xs:complexType>
</xs:element>
<xs:element name="clientVia" vs:help="configuration/system.serviceModel/behaviors/endpointBehaviors/behavior/clientVia">
<xs:complexType>
<xs:annotation>
<xs:documentation>Specifies the URI for which the transport channel should be created.</xs:documentation>
</xs:annotation>
<xs:attribute name="viaUri" type="xs:string" use="optional">
<xs:annotation>
<xs:documentation>A string that specifies a URI that indicates the route a message should take.</xs:documentation>
</xs:annotation>
</xs:attribute>
<xs:attribute name="lockAttributes" type="xs:string" use="optional" />
<xs:attribute name="lockAllAttributesExcept" type="xs:string" use="optional" />
<xs:attribute name="lockElements" type="xs:string" use="optional" />
<xs:attribute name="lockAllElementsExcept" type="xs:string" use="optional" />
<xs:attribute name="lockItem" type="boolean_Type" use="optional" />
<xs:anyAttribute namespace="http://schemas.microsoft.com/XML-Document-Transform" processContents="strict" />
</xs:complexType>
</xs:element>